Friday, August 5, 2011

List of Data Mining / Forcasting Competitions

Kaggle
http://www.kaggle.com/Competitions

TunedIT Solutions
http://tunedit.org/challenges/

Causality Workbench (none current as of Aug 2011)
http://www.causality.inf.ethz.ch/home.php

DARPA's Shredder Challenge (document reconstruction)
Closes December 5, 2011
http://www.shredderchallenge.com/

1st International Competition of Time Series Forecasting
Closes 10th of January 2012
http://www.caos.inf.uc3m.es/~jperalta/ICTSF/

if you know of any more sites offering data mining / forecasting / machine learning competitions please leave a comment!  Thanks.

Friday, May 27, 2011

Free Utilities for a New Machine

NOTE: this is a work in progress, links coming...


Apps

Chrome

Firefox

irfanview

FoxitReader

mRemote

FileZilla

foobar2000

vlc

Skype


System

Wizmouse (scrolling fix) v1.0.0.5.exe

puretext20_x86.zip

notepad++

7zip

Launchy

VirtualBox + Ubuntu

Xming Xserver

putty

SysinternalsSuite

www.magiciso.com/tutorials/miso-magicdisc-overview.htm

ImgBurn

TreeSizeFree

Generic Development Tools

Reflector / JustDecompile www.telerik.com/products/decompiling.aspx

DiffMerge

WinMerge

Expresso Regex

XPathBuilder v2.0.0.4

LINQPad

Screen Ruler

Tortoise SVN

Specific Development Tools

Eclipse

Python

MonoDevelop


Commercial Software

Visual Studio (2008 and 2010)

ReSharper

SQL Server Management Studio

Redgate SQL Compare adn SQL Data Compare

Microsoft Office

Friday, December 10, 2010

Windows hosted virtual private server (VPS) set up

In early 2010 I migrated my websites from shared web host that had begun to suck to a virtual private server (VPS) provided by Web24.

I did this for performance reasons as the shared web host had degraded over the course of several years, and additionally I wanted control over what was installed on the machine just like when at work.

Due to low traffic and budget constraints I'm running the VPS as a web server (IIS), DB server (SQL Server Express, and MySQL) and an SMTP server.  Here are my notes regarding the server installation and configuration so I can do it again fairly quickly if needed.

NOTE: these details were accurate at March 2010, newer versions and service packs are probably available now.

VPS Configuration
Windows Server 2003 R2, 64 bit
IIS 6
1GB RAM
10GB disk space
Parallels Power Panel
Full remote desktop access

General Utilities Installed
7zip, notepad++, Winmerge and Sysinternals Suite

Web Platform Installer
Used the WPI to install: .NET FW 2.0 .NET FW 3.5SP1, MVC 2.0

Database Servers
MySQL:

SQL Server 2008 Express:
SMTP Mail Server
hMailServer:
Web Mail
roundcube (PHP application):
Yep, that's it, too easy really.  Overall I'm pretty happy with Web24 - faster than the old shared hosting, and (almost) complete control over the environment. Only downside is a few more sysadmin tasks.

Next thing to do is switch to Server 2008 with IIS 7.

Entity Framework and LINQ2SQL Links

EF
Migrating from LINQ to SQL to Entity Framework: Eager Loading
http://blogs.msdn.com/adonet/archive/2008/10/07/migrating-from-linq-to-sql-to-entity-framework-eager-loading.aspx

Known Issues and Considerations in LINQ to Entities
http://msdn.microsoft.com/en-us/library/bb896317.aspx


LINQ2SQL

Linq to SQL DataContext Lifetime Management (Rick Strahl)


Python Libraries

Here are some useful Python libraries I've been using for conducting and visualising experiments in machine learning, forecasting and statistical learning. It is updated as I discover new libraries/applications.
Last updated April 2013.

Libraries

numpy - Numerical Python - Numerical Python adds a fast and sophisticated array facility to the Python language. NumPy is the most recent and most actively supported package.

scipy - Scientific Library for Python - SciPy is package of tools for science and engineering for Python. It includes modules for statistics, optimization, integration, linear algebra, Fourier transforms, signal and image processing, ODE solvers, and more.

matplotlib - matplotlib tries to make easy things easy and hard things possible. You can generate plots, histograms, power spectra, bar charts, errorcharts, scatterplots, etc, with just a few lines of code

PyBrain -  Python-Based Reinforcement Learning, Artificial Intelligence and Neural Network Library. PyBrain is a modular Machine Learning Library for Python. Its goal is to offer flexible, easy-to-use yet still powerful algorithms for Machine Learning Tasks and a variety of predefined environments to test and compare your algorithms.

nltk - Natural Language Toolkit — is a suite of open source Python modules, data and documentation for research and development in natural language processing.

PIL - Python Imaging Library adds image processing capabilities to your Python interpreter.

mlpy - Machine Learning PYthon - high-performance Python library for predictive modeling. Makes extensive use of  NumPy to provide fast N-dimensional array manipulation and easy integration of C code. The GNU Scientific Library ( GSL) is also required. It provides high level procedures that support, with few lines of code, the design of rich Data Analysis Protocols (DAPs) for preprocessing, clustering, predictive classification, regression and feature selection. Methods are available for feature weighting and ranking, data resampling, error evaluation and experiment landscaping.

networkx - High productivity software for complex networks - creation, manipulation, and study of the structure, dynamics, and functions of complex networks.

neurolab - a simple and powerful Neural Network Library for Python. Contains based neural networks, train algorithms and flexible framework to create and explore other networks. Pure python + numpy.
Includes: Single layer perceptron, Multilayer feed forward perceptron, Competing layer (Kohonen Layer), Learning Vector Quantization (LVQ), Elman Recurrent network, Hopfield Recurrent network. (Sep 2011)
[Note: after using this library I prefer PyBrain since it feels more stable and better documented. However neurolab has a similar api to the MATLAB NN Toolbox]

scikits.learn - Easy-to-use and general-purpose machine learning in Python. Python module integrating classic machine learning algorithms in the tightly-knit world of scientific Python packages (numpy, scipy, matplotlib).  It aims to provide simple and efficient solutions to learning problems that are accessible to everybody and reusable in various contexts: machine-learning as a versatile tool for science and engineering. Part of SciKits. (Aug 2011)

scikits.* - SciKits (short for SciPy Toolkits), are add-on packages for SciPy, hosted and developed separately from the main SciPy distribution. All SciKits are available under the 'scikits' namespace.
Selected examples that have recent updates and look maintained: (Sep 2011)

Tablib - allows you to import, export, and manipulate tabular data sets. Advanced features include, segregation, dynamic columns, tags & filtering, and seamless format import & export. (Sep 2011)

pyneurgen - Python Neural Genetic Hybrids.  This software provides libraries for use in Python programs to build hybrids of neural networks and genetic algorithms and/or genetic programming. (Sep 2011)

pyml - machine learning in Python. PyML is an interactive object oriented framework for machine learning, it focuses on SVMs and other kernel methods.
Features:
Classifiers: support vector machines, nearest neighbor, ridge regression
Multi-class methods (one-against-rest and one-against-one)
Feature selection (filter methods, RFE)
Model selection
Preprocessing and normalization
Syntax for combining classifiers
Classifier testing (cross-validation, error rates, ROC curves)
(Sep 2011)

Pandas - pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.  R users will be pleased to find this library adopts some of the best concepts of R, like the foundational DataFrame (April 2013)

PyTables - PyTables is a package for managing hierarchical datasets and designed to efficiently and easily cope with extremely large amounts of data. PyTables is built on top of the HDF5 library, using the Python language and the NumPy package. (April 2013)

gensim - realize unsupervised semantic modelling from plain text.  Useful to determine similarity between pairs of documents.  Includes Latent Sematic Analysis and Latent Dirichlet Allocation (April 2013)

ramp - Ramp is a python package for rapid machine learning prototyping. It provides a simple, declarative syntax for exploring features, algorithms and transformations quickly and efficiently. At its core it’s a unified pandas-based framework for working with existing python machine learning and statistics libraries (scikit-learn, rpy2, etc.) (April 2013)

Statsmodels - Statsmodels is a Python module that allows users to explore data, estimate statistical models, and perform statistical tests. An extensive list of descriptive statistics, statistical tests, plotting functions, and result statistics are available for different types of data and each estimator. (April 2013)

Blaze - Blaze is the next generation of NumPy, Python’s extremely popular array library. Blaze is designed to handle out-of-core computations on large datasets that exceed the system memory capacity, as well as on distributed and streaming data. (April 2013)



Applications

IPython - IPython provides a rich toolkit to help you make the most out of using Python interactively. Its main components are:
Powerful interactive Python shells (terminal- and Qt-based).
Support for interactive data visualization and use of GUI toolkits.
Flexible, embeddable interpreters to load into your own projects.
Tools for high level and interactive parallel computing.

Orange - Open source data visualization and analysis for novice and experts. Data mining through visual programming or Python scripting. Components for machine learning. Extensions for bioinformatics and text mining. Packed with features for data analytics.

Friday, November 5, 2010

InfoVis Applications, Libraries

NOTE: This post is a work in-progress


Applications and Libraries

Gephi, App, Java (NetBeans build environment)
Gephi is an interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs. Runs on Windows, Linux and Mac OS X. Gephi is open-source and free.

prefuse, Library, Java
See also: vizster, uses prefuse to visualise social networks.
Prefuse supports a rich set of features for data modeling, visualization, and interaction. It provides optimized data structures for tables, graphs, and trees, a host of layout and visual encoding techniques, and support for animation, dynamic queries, integrated search, and database connectivity. Prefuse is written in Java, using the Java 2D graphics library, and is easily integrated into Java Swing applications or web applets. Prefuse is licensed under the terms of a BSD license, and can be freely used for both commercial and non-commercial purposes.

prefuse flare, Library, ActionScript
Flare is an ActionScript library for creating visualizations that run in the Adobe Flash Player. From basic charts and graphs to complex interactive graphics, the toolkit supports data management, visual encoding, animation, and interaction techniques.

Protovis, Library, Javascript with SVG
Protovis composes custom views of data with simple marks such as bars and dots. Unlike low-level graphics libraries that quickly become tedious for visualization, Protovis defines marks through dynamic properties that encode data, allowing inheritance, scales and layouts to simplify construction.

UbiGraph, Rendering engine, C++,  Linux only
UbiGraph is a tool for visualizing dynamic graphs. The basic version is free, and talks to Python, Ruby, PHP, Java, C, C++, C#, Haskell, and OCaml.

igraph, Library, C++ with Python module and R package
igraph contains functions for generating regular and random graphs, manipulating graphs, assigning attributes to vertices and edges. It can calculate various structural properties, graph isomorphism, includes heuristics for community structure detection, supports many file formats. The R and Python interfaces support visualization.

Graphviz, Tool set
The Graphviz layout programs take descriptions of graphs in a simple text language, and make diagrams in several useful formats such as images and SVG for web pages, Postscript for inclusion in PDF or other documents; or display in an interactive graph browser. (Graphviz also supports GXL, an XML dialect.)


People


Jeffrey Heer

Friday, April 23, 2010

Online Lectures (updated 2012)

Added 2012:


Stanford EE263: Introduction to Linear Dynamical Systems
Introduction to applied linear algebra and linear dynamical systems, with applications to circuits, signal processing, communications, and control systems.
http://www.stanford.edu/~boyd/ee263/index.html
http://www.youtube.com/playlist?list=PL06960BA52D0DB32B


MIT - 18.085 Computational Science and Engineering I
Review of linear algebra, including applications to networks, structures, and estimation, Lagrange multipliers. Also covered are: differential equations of equilibrium; Laplace's equation and potential flow; boundary-value problems; minimum principles and calculus of variations; Fourier series; discrete Fourier transform; convolution; and applications.
http://ocw.mit.edu/courses/mathematics/18-085-computational-science-and-engineering-i-fall-2008/index.htm


Statistical Aspects of Data Mining (Stats 202 - Google/Stanford)
Topics: decision trees, neural networks, association rules, clustering, case based methods, and data visualization.
http://www.youtube.com/playlist?list=PLA40054B49BA80084&feature=view_all
http://www.stats202.com/original_index.html


Added 2011:
Awesome, free, online courses from Stanford offered Oct 2011:

CS229, Machine Learning Course
Broad introduction to machine learning, datamining, and statistical pattern recognition
http://www.ml-class.org/
http://cs229.stanford.edu/
http://www.reddit.com/r/mlclass/

CS221, Artificial Intelligence Course
Covering basic elements of AI, such as knowledge representation, inference, machine learning, planning and game playing, information retrieval, and computer vision and robotics.
http://www.ai-class.com/
http://robots.stanford.edu/cs221/
http://www.reddit.com/r/aiclass

Unsupervised Feature Learning and Deep Learning
Video lectures by Andrew Ng (Stanford, also teaches CS229)
http://openclassroom.stanford.edu/MainFolder/CoursePage.php?course=ufldl


Added 2010:

Stanford - Programming Paradigms (CS107)
Online lectures http://www.youtube.com/view_play_list?p=9D558D49CA734A02
Course notes https://courseware.stanford.edu/info/course/CS107
Lecture summary https://courseware.stanford.edu/pg/courses/lectures/4838

Stanford - Machine Learning (CS229)
Online lectures http://www.youtube.com/view_play_list?p=A89DCFA6ADACE599
Course notes http://www.stanford.edu/class/cs229/
Lecture summary http://www.stanford.edu/class/cs229/schedule.html
Course Materials http://www.stanford.edu/class/cs229/materials.html
Notes
http://amitksaha.wordpress.com/ (229 search)

MIT - Introduction to Algorithms (6.046J / 18.410J)
Online lectures http://www.youtube.com/view_play_list?p=8B24C31197EC371C
Course notes
http://ocw.mit.edu/OcwWeb/Electrical-Engineering-and-Computer-Science/6-046JFall-2005/CourseHome/index.htm
Lecture summary
http://ocw.mit.edu/OcwWeb/Electrical-Engineering-and-Computer-Science/6-046JFall-2005/VideoLectures/
3rd party notes (very good)
http://www.catonmat.net/blog/category/introduction-to-algorithms/

Stanford - Introduction to Robotics (CS223A)

Online lectures http://www.youtube.com/view_play_list?p=65CC0384A1798ADF
Course notes 1 http://www.stanford.edu/class/cs223a/
Course notes 2
http://see.stanford.edu/see/courseInfo.aspx?coll=86cc8662-f6e4-43c3-a1be-b30d1d179743
Lecture summary
http://see.stanford.edu/see/lecturelist.aspx?coll=86cc8662-f6e4-43c3-a1be-b30d1d179743

MIT - Linear Algebra (18.06)
Online lectures http://www.youtube.com/view_play_list?p=E7DDD91010BC51F8
Course notes http://web.mit.edu/18.06/www/
Online lectures download
http://ocw.mit.edu/OcwWeb/Mathematics/18-06Spring-2005/VideoLectures/index.htm
Peteris Krumins' notes
http://www.catonmat.net/?s=MIT+Linear+Algebra&submit=Search

Stanford - Human-Computer Interaction Seminars (CS547)
Course notes http://hci.stanford.edu/courses/cs547/
Online lectures 2006/7 http://www.youtube.com/view_play_list?p=32A089D3E2DFB65D
Online lectures 2007/8 http://www.youtube.com/view_play_list?p=27C635EE182143CE
Online lectures 2008/9 http://www.youtube.com/view_play_list?p=CFE6199B78198411
Online lectures 2009/10 http://www.youtube.com/view_play_list?p=F4A01617834A0FB6


Other interesting lecture series:

Berkeley Computer Science 61A, 001 - Spring 2008:

Stanford Data Mining CS246/CS341 (formerly CS345A)