understanding tools for analyzing data
I wrote a post earlier about how to analyze data using Pandas. In that post, I just introduced some simple Pandas functions to analyze some random data. In this post, I am using real world traffic data from NYC. NYC has made a lot of data available to public recently on this website. There is plenty of quality data for you to play around with. I chose to look at the traffic data which you can download from here.
I performed the analysis using iPython Notebook which I have embedded here.
This is the first post in the WorkTip series that I am starting on Enlist q. I have been working full-time as a developer for about 4 years now and have learnt so much that can only be learnt through experience. I have been lucky enough to be surrounded by some great minds and mentors who have not only taught me how to code but also how to adapt to the business environment. I realize that not everyone is as fortunate and that’s precisely why I would like to share what I have learnt with you! Stay tuned!
For several months after I started work, I was shielded from clients. My manager prioritized the work and assigned it to me. I would then attempt to finish the task and fail (this is when I was still learning q, kdb, linux…pretty much everything). My manager would help me out and with his assistance, I will finish the task and pass it back to him. Done. It was a simple process.
Continue reading “When it comes to estimating effort, be pessimistic”
Some time back, when I was still getting familiar with qsql, I went for an interview at an investment bank. I was interviewed by 4 different people for a total of 2 hours. 3 of the interviewers asked theoretical architecture related questions but the last one was all about writing code. The last interviewer gave me a piece of paper and asked me to write a bunch of qsql statements for different scenarios. One of the questions had to do with parsing two columns that contained first and last name and then joining them together. At that time, I didn’t know how to do that. I didn’t get the job but I realized the importance of qsql and started learning. Later, I would come to realize the importance of functional queries.
Anyways, the point of the story is that you will face many situations during the course of your career where you will have to dissect columns to get your desired output. Yesterday, I had a text file with some data that I needed to dissect and put into a table so I can properly analyze it. In this post, I will discuss how I did that.
Continue reading “Manipulating columns in a table”Here is a new challenge for you – capitalize the first letter of each word in a string.
For example:
q) f "I’m gonna make him an offer he can’t refuse" "I'm Gonna Make Him An Offer He Can't Refuse"
Post your answers in the comment section below and let me know if you have any questions.
P.S. Don’t worry about the case with a trailing space i.e. “Hi how are you “
UPDATE:
Pet usa discount cialis birds can be quite high-priced. A decreased estimations of tadalafil may be the suitable and favorable option for the male suffering from impotence problem. free cialis no prescription Remember, fault is not in suffering, it’s in hiding. cute-n-tiny.com vardenafil price Certain fruits have been recognized to have much higher nutritional substance and antioxidant properties canadian levitra online than others.Great solutions everyone! We have quite a variety this time.
Here are all the solutions:
David - {@[x;0,1+where x=" ";upper]} (a more complete version in comments)
Rolf - {@[x;where" "=" ",x;upper]}
Showvik - {ssr[c;” ?”;upper]}
Bill - {s:” ” vs trim x;” “sv .[s;(til count s;0);upper]}
Me - {" " sv .[b;((til count b:" " vs x);(0;0));upper]}
David and Rolf have similar solutions. They used @ to modify the string.
Showvik has a more simplistic approach. He used ssr function to search and replace parts of the string. The only issue is that it won’t cover the first word but that wasn’t specified as a requirement.
Bill and my solutions are both very similar (pretty much the same). We break down the string first and then modify at depth using . operator and then reconstruct the string. It’s a little sophisticated and it’s simpler to use @ approach.
Thanks everyone for participating!
Recently, I have been playing around with Python and its data analysis library – Pandas, which is built on another library called NumPy. Most of you have probably heard of Python (if not then I don’t know what’s wrong with you. Get out there and make some programmer friends or read some blogs). Python has been in existence for a while (since 1991) though it has gained a lot of traction just recently. A lot of startups are into Python. Great thing about Python is that you can use it as a functional language or OOP language. I am more of a functional guy and prefer writing straightforward code. Moreover, I am into data analysis…as opposed to…lets say…designing ugly GUIs.
Anyways, if you want to do data analysis with Python, you must use Pandas. I mean you could use other methods but then you will face serious issues and probably not be good at your job. Pandas is quite pleasant. Coming from a kdb background, I missed seeing data in tabular format. Pandas displays data in dataframes (tables) and allows you to perform operations on columns just like kdb.
Continue reading “Introducing data analysis with Python and Pandas”Few weeks ago, I quit my first job as a q developer and moved to another company. My first job was where I was introduced to q/kdb+ and time-series data. If it wasn’t for that job, I wouldn’t be writing about q on this blog. However, it was time to move on and so I did. While I will still be working with time-series data, I won’t be actively coding in q. I will be using a vendor product called OneTick which is similar to kdb in functionality and comes with many features built-in. Advantage of that is you have a working solution out of the box. On the other hand, it lacks flexibility that comes hand-in-hand with kdb.
Besides learning OneTick, I will also be learning other languages such as perl and python. I have started learning them already and it’s quite interesting programming in other languages when you are coming from a q background. Maybe I will write a post in near future to compare the languages.
But check my drugshop viagra for females it doesn’t mean that one doesn’t need precautions or prescriptions. Like viagra 25 mg, icariin, the active compound in Epimedium, inhibits the activity of PDE-5 allowing enhanced blood circulation in penis tissues. Kamagra causes an buying cheap cialis energetic production of cyclic GMP resulting the pennies to produce erection. These kinds of diet have some far reaching positive effect on the human body. buy cheap sildenafil why not try these out I am not abandoning q/kdb. I fully understand the importance and potential of the database and will not be giving up on it. It’s a valuable skill that I intend to take full advantage of. I will continue to develop in q on the side…just not as much. What this means for the blog is that I will be writing about other stuff besides q as well. I will be writing about coding and life as a developer.
Stay tuned for some great new content!
In one of my previous posts, I discussed functional form of select, update, exec and delete. If you remember, the general form of a functional select is ?[t;c;b;a] where t is the table, c is for constraints, b for by clause and a for aggregation. Most of the time, you will see only these 4 arguments but functional select can actually have up to 6 arguments!
The full form looks like this:
?[t;c;b;a;n;i]
where n stands for number of rows and i for specific rows you want to extract.
Let’s look at some examples!
Check out this great chat client written in q/html with less than 50 lines of code. The author, Stephen Peck, describes it as “Persistent group chat in <50 lines using kdb+/k/q web sockets and JS”. You only need a working version of q/kdb and a browser to get it started. You can find the code here.
Here is a screenshot of what it looks like in action:
Isagenix Australia Cleanse for Life Cleanse for Life is a great detox plan to follow if you want to eliminate impurities and toxins naturally. cialis online At levitra without rx present, the most common prostate disease is prostatitis. In earlier times when the world wasn’t crazily busy – this could have been possible but today when we fall short of time even to look after genital health during ED? Erectile dysfunction, very ordinary male sensual malfunctioning can last for a one or two week recommended cialis 5 mg and if you experience this condition, you must seek immediate medical attention. In that situation, the veins and arteries of generic tadalafil cheap the organ and the stamina and strength thousand times. My favorite thing about this project is not how simple it is but how it uses q to do something that’s not related to data/databases. Any project I have worked on has had to do with databases and why shouldn’t it? This is what the language was designed for. Seeing q being used to do something different is very interesting and really makes you think what else you can use q for (I really shouldn’t be ending my sentences with prepositions).
Anyways, check out the code. Full credit goes to Stephen Peck! Good job!
Sorry for the delay in coming up with a new challenge. It’s been a busy few weeks. Last challenge was about partial sort. This week’s challenge will be about reversing words in a phrase. It’s a common challenge and many of you have probably seen it already.
Given a string with some words, produce a string in reverse order.
q)f "you must be outta your damn mind" "mind damn your outta be must you"Also, they should not be taken with grape fruit and other ED medicines to let it work well and stay away from bad habits. cheap levitra india icks.org It provides planned and much cost effective solution for the cure viagra free order of ED. Others will simply ask you for the canadian viagra sales. Making discover for source buying cialis in canada use of Kamagra Fizz a male may respond to erotic activation within a sex experience.
P.S. Bonus points for getting the reference!
Post your answers as comments.
Continue reading “Challenge#2 Reverse words in a phrase”In one of my previous posts, I talked about functional forms of select, exec, update and delete. I also briefly touched on the importance of functional queries and why you should get used to them as much as possible if you are serious about q.
In this post, I will talk about functional forms of amend (@ and .) which I think are even more important than select/update. You can use amend to modify lists/dictionaries/tables via a custom or built-in function. A major advantage of amend is its ability to modify both in-memory and on-disk tables.
There are two versions of amend: @ and .@ is for higher-level modification where as . modifies nested elements.