Thursday, December 6, 2018

Python Tutorial Series - Set Up

I'm working on learning Python as part of learning about Data Science, so in the process of doing that, I thought I'd put together a series of Python tutorial blog posts. Python is supposed to be easy to learn, so we'll see.

In order to use Python on your local machine, it needs to be downloaded from here. There are 2 major versions available. The latest is 3.7.1. There is also a version 2.7.15. I'm going to focus on version 3, which is the result of cleaning up version 2 without regards to backwards compatibility. Version 2 will not have any new major releases, but it's still in use by a lot of people. In fact, an online course I tried taking still used version 2. It is possible to install version 2 and version 3 on the same computer though. The trick to getting them both to work from the command line is to rename the executable of one version a different name than the other version. For example, I renamed python.exe for version 2 python2.exe. In a Windows command prompt, I can run version 3 by typing python. I can also run version 2 by typing python2 in the command prompt. Both versions need to have their directories added to the environment variable PATH setting.

For version 3, add the following to the environment variable PATH setting:

C:\your path to python3\Scripts\

C:\your path to python3\

For version 2, add the following to the environment variable PATH setting:

C:\your path to python2

To code in python, Anaconda is a popular data science tool which comes packaged with Python. There are also online Python editors, such as https://www.onlinegdb.com or https://repl.it/repls/SoftRashSearchengine.



More Python







Thursday, November 27, 2014

Columnar Database

I recently read about an interesting database concept that uses a columnar database, which is where data is stored in columns instead of rows.
Apparently this approach has several advantages over a row based database. A columnar database has better performance when a large number of queries are run at the same time. Joins and aggregation are faster as aggregate function results are incrementally calculated. Columnar databases can be compressed which eliminates the storage of multiple indexes, views and aggregation. Columnar databases also load data quickly. Each column is built in one pass and stored separately, so multiple threads can load different columns at the same time. Also, since columns are stored separately, data can be added and removed without updating the system after each change.
There are disadvantages to a columnar database though. It doesn't work for every case. Converting a row based database into a columnar database can be slow for large datasets. Incremental loads can be slow. Uncompressing data for reading can be slow. There are structural limitations, such as some columnar databases require the same primary key on all tables. These imitations may make it difficult to expand the system in the future. Columnar databases are also meant for large datasets and may not be suitable for small datasets.

This post was brought to you by your friends at MemSQL.

Thursday, May 23, 2013

How I Learned PHP

Learning PHP was on my list of things to learn for a while. Two years ago, I finally started.
I learned PHP by taking two online courses at ed2go.com. They were called: Introduction to PHP and MySQL and Intermediate PHP and MySQL. They were fairly good at getting you started. The first course guides you through the creation of a recipe website with it's own content management system (CMS). The second course guides you through the creation of an online store.
I also purchased a few books:

PHP Solutions: Dynamic Web Design Made Easy

PHP and MySQL Web Development (4th Edition)

Programming PHP

Codeacademy.com offers free online courses, including PHP. They are actually pretty good.

Php.net is the official PHP documentation site.

If you have the foundation of knowing another programming language, PHP is pretty easy to learn. For me, the hardest part is remembering to include the $ sign.


More PHP Resources


Friday, May 3, 2013

Can't Create New Folder Windows 7 Fix

A month or so ago, I noticed the strangest thing on my laptop. I couldn't create a new folder from within Windows Explorer. However, I could create a new folder using the mkdir command from within a command prompt window. I have no idea what happened to cause this. I'm guessing it was caused by an installation or a windows update.
Today I finally took the time to see if I could fix it. Luckily, I did. After some Google searching, I decided to give this suggestion a try: http://social.technet.microsoft.com/Forums/en-US/w7itprogeneral/thread/97de8a2a-12f2-4381-a409-a78f4ae551cf


which states:

First, please use System File Checker tool to troubleshoot this issue.

If the issue persists, import the following registry key and see if the issue can be resolved.

First, please refer to this knowledge base (KB) article to back up the registry key. After backing up the registry, please follow the steps below to import the registry:

1. Click Start, type Notepad in the Search bar and pres ENTER.
2. Copy the following commands and then paste them into the open Notepad window.

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\CLSID\{11dbb47c-a525-400b-9e80-a54615a090c0}]
@="CLSID_ExecuteFolder"

[HKEY_CLASSES_ROOT\CLSID\{11dbb47c-a525-400b-9e80-a54615a090c0}\InProcServer32]
@="ExplorerFrame.dll"
"ThreadingModel"="Apartment"

[HKEY_CLASSES_ROOT\Directory\Background\shellex\ContextMenuHandlers\New]
@="{D969A300-E7FF-11d0-A93B-00A0C90F2719}"

[HKEY_CLASSES_ROOT\Folder]
@="Folder"
"EditFlags"=hex:d2,03,00,00
"FullDetails"="prop:System.PropGroup.Description;System.ItemNameDisplay;System.ItemType;System.Size"
"ThumbnailCutoff"=dword:00000000
"TileInfo"="prop:System.Title;System.PropGroup.Description;System.ItemType"

[HKEY_CLASSES_ROOT\Folder\DefaultIcon]
@=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,25,\
00,5c,00,53,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,73,00,68,00,\
65,00,6c,00,6c,00,33,00,32,00,2e,00,64,00,6c,00,6c,00,2c,00,33,00,00,00

[HKEY_CLASSES_ROOT\Folder\shell]

[HKEY_CLASSES_ROOT\Folder\shell\explore]
"MultiSelectModel"="Document"
"BrowserFlags"=dword:00000022
"ExplorerFlags"=dword:00000021

[HKEY_CLASSES_ROOT\Folder\shell\explore\command]
@=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,25,\
00,5c,00,45,00,78,00,70,00,6c,00,6f,00,72,00,65,00,72,00,2e,00,65,00,78,00,\
65,00,20,00,2f,00,73,00,65,00,70,00,61,00,72,00,61,00,74,00,65,00,2c,00,2f,\
00,65,00,2c,00,2f,00,69,00,64,00,6c,00,69,00,73,00,74,00,2c,00,25,00,49,00,\
2c,00,25,00,4c,00,00,00
"DelegateExecute"="{11dbb47c-a525-400b-9e80-a54615a090c0}"

[HKEY_CLASSES_ROOT\Folder\shell\open]
"MultiSelectModel"="Document"
"BrowserFlags"=dword:00000010
"ExplorerFlags"=dword:00000012

[HKEY_CLASSES_ROOT\Folder\shell\open\command]
@=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,25,\
00,5c,00,45,00,78,00,70,00,6c,00,6f,00,72,00,65,00,72,00,2e,00,65,00,78,00,\
65,00,20,00,2f,00,73,00,65,00,70,00,61,00,72,00,61,00,74,00,65,00,2c,00,2f,\
00,69,00,64,00,6c,00,69,00,73,00,74,00,2c,00,25,00,49,00,2c,00,25,00,4c,00,\
00,00
"DelegateExecute"="{11dbb47c-a525-400b-9e80-a54615a090c0}"

[HKEY_CLASSES_ROOT\Folder\shellex]

[HKEY_CLASSES_ROOT\Folder\shellex\ColumnHandlers]

[HKEY_CLASSES_ROOT\Folder\shellex\ColumnHandlers\{F9DB5320-233E-11D1-9F84-707F02C10627}]
@="PDF Column Info"
[HKEY_CLASSES_ROOT\Folder\shellex\ContextMenuHandlers]

[HKEY_CLASSES_ROOT\Folder\shellex\ContextMenuHandlers\BriefcaseMenu]
@="{85BBD920-42A0-1069-A2E4-08002B30309D}"

[HKEY_CLASSES_ROOT\Folder\shellex\ContextMenuHandlers\Offline Files]
@="{474C98EE-CF3D-41f5-80E3-4AAB0AB04301}"

[HKEY_CLASSES_ROOT\Folder\shellex\ContextMenuHandlers\{a2a9545d-a0c2-42b4-9708-a0b2badd77c8}]

[HKEY_CLASSES_ROOT\Folder\shellex\DragDropHandlers]

[HKEY_CLASSES_ROOT\Folder\shellex\DragDropHandlers\{BD472F60-27FA-11cf-B8B4-444553540000}]
@=""
[HKEY_CLASSES_ROOT\Folder\shellex\PropertySheetHandlers]

[HKEY_CLASSES_ROOT\Folder\shellex\PropertySheetHandlers\BriefcasePage]
@="{85BBD920-42A0-1069-A2E4-08002B30309D}"

[HKEY_CLASSES_ROOT\Folder\shellex\PropertySheetHandlers\Offline Files]
@="{7EFA68C6-086B-43e1-A2D2-55A113531240}"

[HKEY_CLASSES_ROOT\Folder\ShellNew]
"Directory"=""
"IconPath"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,\
74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,73,\
00,68,00,65,00,6c,00,6c,00,33,00,32,00,2e,00,64,00,6c,00,6c,00,2c,00,33,00,\
00,00
"ItemName"="@shell32.dll,-30396"
"MenuText"="@shell32.dll,-30317"
"NonLFNFileSpec"="@shell32.dll,-30319"

[HKEY_CLASSES_ROOT\Folder\ShellNew\Config]
"AllDrives"=""
"IsFolder"=""
"NoExtension"=""


3. After pasting the above commands, please click File on the menu and click Save.
4. Type in NewFolder.reg as the file name, select to save the file on the Desktop. Click Save.
5. Go to the Desktop, and double click NewFolder.reg. A dialog box will pop-up saying "Are you sure you want to add the information in fix.reg to the registry?". Click Yes to confirm.

Note: Administrative privileges are required to perform the above steps. Please click "Continue" when the User Account Control dialog-box appears.


I did everything as stated above:

1. Open up a command prompt window by right clicking Run As Administrator and type in: sfc /scannow - No problems were found for me.

2. I backed up the registry by going to Start and typing in regedit. Click on Computer in the left panel. Go to File, Export, type in a file name and save.

3. Open up Notepad and copy all of the above from Step 2 in the article suggestion. Follow Steps 3, 4, and 5 from the article suggestion. I did get a pop up box similar to Step 5, but it was worded differently. I then got another pop up box with an error message that other processes were in use and the key couldn't be written.
I then tried creating a New Folder in Windows Explorer by right clicking. The New Folder option still wasn't there, so I clicked on the New Folder menu item. Guess what? A new folder was created! I then tried the right click and New Folder is now showing as an option.

No idea what caused the problem of not being able to create a New Folder from within Windows Explorer, but that was the fix that worked for me. BTW, my laptop is running Windows 7.


More Windows 7 Resources




Thursday, February 21, 2013

What To Do Frozen Windows Update

During the day, I refuse to reboot my laptop unless it's absolutely necessary because I never know when Windows will try to install an update. Installations normally finish pretty quick and I can continue, but a few months ago I rebooted and the installation just sat there forever. The screen said it was installing x out of y updates with blinking periods. I wasn't sure if it was doing something or not, so I left it. After a few hours I was really starting to wonder. I finally ended up researching what to do on the internet using my other laptop.
The same thing happened to me yesterday. I let the install go for 5 hours before interrupting it.
What I did was hold the power key down for about 10 seconds. The laptop turned off. I pushed the power button again to turn it back on and chose startup in Safe Mode. Safe Mode took care of whatever the issue was and booted into Windows. I then restarted into normal windows mode. Things seem ok except whenever an app called 4shared Desktop tries to start, it gives an error message. No big deal since I don't use it anyways.
5 hours is a good length of time to wait, 2 - 3 hours minimum, before killing a Windows update. One site I read even said to wait 30 minutes. I'd rather wait a little longer just to be sure. Killing a Windows update when it's actually doing something might cause more harm than good.
BTW, my laptop is an HP Pavilion dv5 running Windows 7.





Wednesday, February 20, 2013

Facet Search Implementation Tips

I recently added a facet search capability to a website. It's really cool! Facet search allows the user to apply filters to a collection of data in order to narrow down the results. Implementation is more complicated than it initially sounds.
Assuming the data is stored in a database, one way to implement facet searching is to repeatedly call the database with a new query whenever a filter is added or removed. This way can get complicated, but it's possible. It seems like it may be a little slow too. Here are a few links with details on how to implement it: http://stackoverflow.com/questions/8300675/php-navigation-with-filters http://ivaldi.nl/2012/05/seo-friendly-faceted-search-1/
The other way to implement facet searching is to load all of the data once when the page loads. When a filter is activated or deactivated, the filter is applied to the stored data and the results updated. Loading all of the data at once may be a little slow, but once the data is all loaded the filtering goes pretty quick. You can hook in ajax, so that when the filter is activated or deactivated, the results immediately update without a page reload. The site I was adding this to is pretty complex and to reload the page each time wouldn't be the greatest user experience. There were a couple of examples I followed for the implementation. The main one being: http://eikes.github.com/facetedsearch/

Other helpful examples were:
http://proj.2ality.com/facetator/


The facet menu had 3 facets. One was a list of checkboxes. The other two were a series of links where only one link in each facet could be active at a time.
I made a template file for the general outline of the facet menu, a script file to control features of the facet menu itself, such as hide a facet when only one choice is allowed and updating a current selections div. It sat on top of the facetedsearch example script. That file needed a few changes in order to accomodate the data being filtered and to blend the results into the way the page was displaying the results.
I also changed the way the counts were displayed. I took out the + and found the counts for each option based on the filtered data. That was a little tricky, but worked out good.
Another key piece in making the facet search run smoothly is to add a hash tag at the end of your url with parameters for the currently set filters. ex: www.abc.com/#x=1&y=2
Doing this ensures the back/forward buttons navigate smoothly. Some helpful links about that are:

http://blog.mgm-tp.com/2011/10/must-know-url-hashtechniques-for-ajax-applications/
I ended up using this plugin for Internet Explorer 7 to work properly: http://benalman.com/projects/jquery-hashchange-plugin/

For browsers supporting history.pushback, you can just do:
window.history.pushState({path : pageurl}, '', pageurl); to add the hash tag.

For browsers not supporting history.pushback, you do:
window.location.hash = filter_str;

To make it work, you do:
var originalHash = window.location.hash;
if (window.history && window.history.pushState) {
window.addEventListener("popstate", function(e) {

var newHash = window.location.hash;
if(newHash != originalHash) {
//get the filters from the url, apply them & update the results
}

originalHash = newHash;
});
}
For Internet Explorer 7:
$(function(){
$(window).hashchange( function(){
var newHash = location.hash;
if(newHash != originalHash) {
originalHash = newHash;
//get the filters from the url, will need to compare them against current filter,
apply them if different, update the results
}
})

// Since the event is only triggered when the hash changes, we need to trigger
// the event now, to handle the hash the page may have loaded with.
$(window).hashchange();
});

For Internet Explorer 8+:
$(window).bind('hashchange', function() {
var newHash = window.location.hash;

if (newHash != originalHash) {
//get the filters from the url, will need to compare them against current filter,
// apply them if different, update the results
}

originalHash = newHash;
});

Facet search menus are generally located on the left, sometimes at the top, not usually on the right. Example sites that implement facet search menus include: Old Navy, Walmart, Amazon, Toysrus, Best Buy, Ebay and Kmart.


More JavaScript


Wednesday, December 19, 2012

Data Storage Backup Plan

I've been doing some final Christmas shopping the past few days. I hate shopping. It takes me forever to figure out what to buy, and then I spend lots of time comparing brands and prices. I'll be so glad when Christmas is over, and I can get back to normal.
My brother just wants money and gift cards, so at least he was easy. He wants to buy a laptop since his desktop died. To go along with the cash, I purchased a Kingston Digital 16 GB USB 2.0 Hi-speed Datatraveler Flash Drive DT101G2/16GBZ. It's $9.99 at Amazon.com. Pretty good price for 16gb of data storage. I got one for myself too. I really need to come up with a better backup system for my laptop. If anyone can recommend free backup software, please leave me a comment.
I'm paranoid about making backups of individual files when writing code, but when it comes to making backups of my hard disk, I need improvement. I just don't take the time to do it, so I need something automated. A 16gb flash drive should be a good size for storing what I need to backup.
There are companies out there that specialize in backup systems. One company is called Nexsan. They are one of the leaders in data storage production. Their storage systems have built in features to backup and restore. They have an E-series storage system that is ideal for mid-sized companies.
Backing up your data is crucial. The hard drive on my old work laptop suddenly went one day and nothing could be recovered. That really sucked, but I didn't really lose anything because I had backups. Good thing.