Matlab

[DominiConnor]

I'm interested in the % of MFE students who use Matlab, any ideas ?

[Peik Looi]

We use C++ and Excel VBA in our assignments.

[Bridgett]

some current students do use MatLab though, but I'm not sure about the %.

[Andy]

I'm interested in the % of MFE students who use Matlab, any ideas ?
It's interesting you asked because I notice many shops increasingly use Matlab for the models development. They told me that they chose Matlab for the shorter development time. They can convert to C++ using Matlab compiler.

What is your take on Matlab, Dominic ?

[DominiConnor]

Matlab is indeed getting a good share of model development. Banks vary a lot in how they see it as adding value to a Cv. Zero if they don't use it, and some see it as rather like someone bragging that they are "really good at doing charts in Excel".
But there is a market for people who know it well and can do finance.
We are seeing the emergence of a market for it as a distinct skill like C++, VBA etc.

But like anything else you put on your CV, it' ,laying down a challenge to be asked hard questions on it.
We do have a requirement going for a serious Matlab guru at a tier one bank, though I set an expectation with them, that "someone will turn up before Christmas, probably". Deep Matlab skills are quite rare. So far we've found precisley one newbie at this level, and the fight over who hired him was nasty enough that I found it appropriate to give him personal counselling on the issue.

We see Matlab as promising enough that we're starting to build a relationship with the firm itself.

[tigergb]

we use matlab quite often for daily research because of its built-in function and convenience, but switch to C++ for formal use.

[Andy]

While doing some research on which software is more suited for large scale data analysis, I came across this discussion which I found useful. It compares Matlab vs SAS

I agree, MATLAB is a great product, but in my opinion it completely depends on what your doing. If your doing any "mathematical" type problems (i.e. large scale simulations, equation solving, etc ...) I think MATLAB is far superior to SAS. The language is relatively easy to learn as is a much more "obvious" programming language than SAS. For example, if you want to create an nxn matrix of zeros, the MATLAB command is zeros(n,n) ... or for an nxn identity matrix the command is eye(n). Plus if you want your plots to look good, MATLAB plots are definetly the way to go. (Unless you buy the expensive SAS/GRAPH add-on)
However, if your doing any "statistical" problems (i.e. predictive modeling, statistical analysis, etc ...) I think SAS is far superior in these applications. SAS also does very well with massive datasets and has a lot of great built in procedures.
So, depending on what your doing, MATLAB and SAS both have there advantages and disadvantages. They're both expensive, but if your companies paying for them ... I say get both !
Hope that helps
Eric B
John,

If you simply compare stat toolbox in matlab with stat/SAS, I totally
agree with you that SAS is much better than matlab. However,

1) it is much easy to do generic programming and write your own functions in matlab. There is no comparison between writing SAS macro and writing functions in matlab.
2) it is very easy to create high-quality chart in matlab. The chart quality in matlab is comparable with the one created by S+/R.
3) if you are using neural net, the functions provoded by Eminer/SAS is not even close to the neuralnet toolbox in matlab.
4) in tems of price, I think matlab is much cheaper than SAS.
I have to chime in and agree with Eric B. Whenever I have a complex problem involving matrix algebra, I tend to turn to MatLab, but I'm far more likely to use SAS for everyday statistical problems. The
addition of anovan (and its ability to handle nested designs) is a big recent improvement to MatLab's toolbox, but the statistical tools in SAS still far outsurpass those in MatLab. SAS is also better at automation as it seamlessly integrates with
.Net architecture -- something MatLab is still working on. MatLab's programming language is nevertheless easy to use and understand --if you have a strong matrix algebra background -- so like Eric, I say get
both.

Cheers! John

[Jonathan]

SAS is deeper than just a programming language, the documentation for the usages and functionalities are endless. At work, we occasionally use SAS/IML (Interactive Matrix Language). If you want a identity matrix or zero matrix, than its I(n) and j(n,n,0). There are some guys here 10 years and still learning new things with SAS.

[Andy]

I agree that SAS document is just way too much for one to master. Matlab is much easier to learn. Also, Amazon is selling the SAS documents that you can download from SAS website for free ;)

I like the SAS PROC pretty much.

So Jonathan, how did you get your dataset ? Is it local or remotely ?
I couldn't get the work machine to connect to our SQL server so I'll bring my laptop over and try again tomorrow.

[Jonathan]

Well, I access the Oracle DB in Unix using Putty. The SAS code has some preliminary commands to get access the datasets, (its typically a libname statement with pswd and usrname). If there is too much traffic on the system then I just export the datasets that I need to my PC using the WinSCP tool.

[Andy]

Nice input.
I did SAS/ACCESS with libname and pass-through but it complained about ODBC driver not found. The database I'm dealing with is 100GB in size and updated everynight so download it locally is not an option.

SAS/CONNECT to a SAS server to run batch job would be nice but we don't have a SAS server set up yet. I'll try to connect it from my laptop tomorrow and see what happen.

In any case, I probably talk to you soon. I'm interested in learning more about the SAS tricks you know ;)

[Yuriy]

SAS is very powerful in terms that you can do virtually everything. Although, it might not be the most efficient way :)
I would vote for SAS, but keep an eye on S-Plus as well.
Actually, we could try to compare SAS and S-Plus, rather than SAS and Matlab, since, in my opinion, SAS and Matlab are designed to do different things.

[joe_bradley]

While you're at it, toss in R. GNU R is the free/open-source equivalent to S-plus and I hear it's pretty competitive in terms of useful functionality. In any case, for any sort of stuff I want to do at home, I suppose I'll -have- to learn R since the price is right.

[Andy]

In any case, I probably talk to you soon. I'm interested in learning more about the SAS tricks you know ;)
After talking to Jonathan, it turns out that my SAS does not have the license to use SAS/ACCESS interface to SQL server.
Running this will display all the licensed components installed on your machine
proc setinit;
run;

[Andy]

While looking for a comparison between R/SPlus/SAS/Matlab, I came across this from wilmot
http://www.wilmott.com/messageview.cfm?catid=8&threadid=54251

It's from a NYU MF student. There are some good info, the least of it is that they use Excel/VBA to do hw at NYU ;)

[Yuriy]

By the they, are there any thoughts about having a course in SAS/S-Plus? Half semester in SAS, half in S-Plus.
Part of the course could be devoted to learning SAS and databases, which seems to be a very important.

[kean]

I would consider SPlus and R are in the same category, except S-Plus needs a commercial license. In addition, S-Plus is more powerful when you add finmetric (a separate module) for financial modeling.

I do not have much experience with Matlab. I use Mathematica.

SAS is very powerful for statistical analysis. I think data mining in SAS is the best due its powerful Enterprise Date Miner. If SAS is a superset then S-Plus is closing the rank.

[Andy]

I'm gonna try to see if Matlab can handle huge amount of data tomorrow. I read a bit about all different modules in Matlab and it seems very capable.

[kean]

I plan to pick up Matlab too. Please update us your success with Matlab.

I'm gonna try to see if Matlab can handle huge amount of data tomorrow. I read a bit about all different modules in Matlab and it seems very capable.

[John]

I would consider SPlus and R are in the same category, except S-Plus needs a commercial license. In addition, S-Plus is more powerful when you add finmetric (a separate module) for financial modeling.

I do not have much experience with Matlab. I use Mathematica.

SAS is very powerful for statistical analysis. I think data mining in SAS is the best due its powerful Enterprise Date Miner. If SAS is a superset then S-Plus is closing the rank.

S-Plus' finmetrics, IHMO, is a great tool to learn time series analysis.

[John]

By the they, are there any thoughts about having a course in SAS/S-Plus? Half semester in SAS, half in S-Plus.
Part of the course could be devoted to learning SAS and databases, which seems to be a very important.

I think this would make a great elective.

Regarding these two, SAS is tailored-made for handling large set of data. It is used, e.g., to perform mortgage/MBS prepayment forecast. S-Plus, on the other hand, is more like Matlab, in that it allows one to do quick-and dirty modeling prototyping and development, in addition to statistical analysis.

[Andy]

Someone mentioned to me that most hedge funds use Matlab. I'm not sure if it's true or any article regarding this. I've seen a few examples agreeing to this but it's certainly not indicative of the whole industry.

[John]

Someone mentioned to me that most hedge funds use Matlab. I'm not sure if it's true or any article regarding this. I've seen a few examples agreeing to this but it's certainly not indicative of the whole industry.

S-Plus is also very popular among the HFs, as matter of fact its finmetrics is, sh...., perfect for stat arb trading...

[Andy]

S-Plus is also very popular among the HFs, as matter of fact its finmetrics is, sh...., perfect for stat arb trading...
I'll ask the stat arb guys to see what they use.

John,
How long did you wait to download the SPLUS student version ? i don't think I ever got the email from them to download.

[John]

I'll ask the stat arb guys to see what they use.

John,
How long did you wait to download the SPLUS student version ? i don't think I ever got the email from them to download.

I was given the download & registration information right after I have faxed my student credential. It was a quick turnaround as far as I remember.

However, it does not come with the optional finmetrics module, which was a bit of dissappointment. Perhaps you can ask your company to purchase a license, it is definitely worth it. (I have purchased the old student version of S+ 6.1 (no license expiration) and and s+finmetrics 1.0 from a while ago and it works like a charm)

[Andy]

I looked at the SPLUS just now and they have several modules and I take it only the FinMetrics module is of any use to us ?

I can find out about the software part. It's never been a problem for me ;)
Can you give us some examples of where you used SPlUS ? is your data on a remote server or local file ? I see they have SPLUS server as well but I don't think i will ever get to that, mostly will have to use local computer to do number crunching.

[John]

I looked at the SPLUS just now and they have several modules and I take it only the FinMetrics module is of any use to us ?

I can find out about the software part. It's never been a problem for me ;)
Can you give us some examples of where you used SPlUS ? is your data on a remote server or local file ? I see they have SPLUS server as well but I don't think i will ever get to that, mostly will have to use local computer to do number crunching.

data could be residing anywhere, S-Plus is very flexible in this regarding. The following is from S-Plus 6.1 online help file:


Import File Type

Select the format of the file you wish to import:
<LI class=mvd-P-Bullet>ASCII files (space-, whitespace-, tab-, comma-, and user-defined-delimited)
<LI class=mvd-P-Bullet>dBase files (.DBF II, II+, III, IV files)
<LI class=mvd-P>Epi Info files (.rec)
<LI class=mvd-P>Fixed Format ASCII files (.fix, .fsc)
<LI class=mvd-P-Bullet>FoxPro files (use same import filter as dBase files)
<LI class=mvd-P-Bullet>Gauss (.DAT files - automatically reads the related DHT file)
<LI class=mvd-P-Bullet>Lotus Worksheets (.WKS,.WK1,.WK3,.WK4 and.WRK)
<LI class=mvd-P-Bullet>Matlab (.MAT)
<LI class=mvd-P-Bullet>Microsoft Access files (.MDB)
<LI class=mvd-P-Bullet>Microsoft Excel (versions 2.1 through Excel ‘97 .XLS* files)
<LI class=mvd-P>Microsoft SQL Server files (.sql)
<LI class=mvd-P>Minitab Workbook files (.mtw)
<LI class=mvd-P-Bullet>Paradox Databases (.DB, through Version 7)
<LI class=mvd-P-Bullet>Quattro Pro Worksheets (.WQ1)
<LI class=mvd-P-Bullet>SAS files
<LI class=mvd-P-Bullet>Windows/OS2 (.SD2)
<LI class=mvd-P-Bullet>Version 7/8 - Windows, Sun, and Linux (.SAS7BDAT, .SD2)
<LI class=mvd-P>HP IBM and Sun UNIX (.SSD01)
DEC UNIX (.SSD04)
<LI class=mvd-P-Bullet>SAS Transport files (version 6.x. TPT, XPT). Some special export options may need to be specified in your SAS program. We suggest using the SAS Xport engine (not PROC CPORT) to read and write these files.
<LI class=mvd-P>SigmaPlot files (.jnb)
<LI class=mvd-P-Bullet>SPSS files (.SAV)

Windows OS/2, DEC, HP IBM, Sun UNIX, DEC UNIX
<LI class=mvd-P-Bullet>SPSS Export files (.POR)
<LI class=mvd-P-Bullet>S-PLUS (.SDD)
<LI class=mvd-P-Bullet>STATA files (.DTA Versions 2.0 and higher)
Systat files (double precision, .SYD, or single precision, .SYS)
Note

The following ODBC file formats can be imported by selecting File http://quantnet.org/forum/mk:@MSITStore:C:\Program%20Files\Insightful\splus61\cmd\GUI.CHM::/arrow5.gif Import Data http://quantnet.org/forum/mk:@MSITStore:C:\Program%20Files\Insightful\splus61\cmd\GUI.CHM::/arrow5.gif From ODBC Connection:
<LI class=mvd-P>Informix (.ifx)
Oracle databases (.ora)
Note: All imports from Oracle files are done using ODBC, so the various ODBC components must be properly installed. When exporting to Oracle, table names and column names must be in UPPERCASE.

SYBASE SQL server (.syb)

[Andy]

Thanks for bringing this up, John. I overlooked SPLUS. Maybe it has exactly what I need and lack from both SAS, Matlab.

Read this on their site
Trading Strategies

Insightful Risk Solutions

In fast-paced markets, leading traders require rapid access to the right information to make important market-driven decisions. Inundated with volatile financial data, traders require answers they can rely upon quickly and constantly. Insightful’s S-PLUS and econometric libraries, S+FinMetrics and S+NuOPT provide rigorous technical and fundamental analysis using sophisticated models to predict future behavior.

You need advanced risk analysis of current and future exposure to optimize your financial performance. When you partner with Insightful you can quickly implement, monitor and adjust proprietary trading strategies to meet dynamic market conditions and improve your risk/return profiles. Insightful has provided advanced analytics and consulting to the world’s top financial institutions for20 years. We can help you create exactly the solution that your organization requires with our modern, cutting-edge methods and flexible, extensible solutions that can be easily integrated with your existing systems.

Insightful Solutions deliver:
World-class time-series analysis and forecasting:
Multivariate GARCH models
State-space time-series modeling
Co-integration models
Trading signal generation
Rolling window analysis
System Back-testing framework
State of the art risk management capabilities
Multi-factor models
Robust statistics
Extreme-value distributions

The part about trading signal generation is of interest to me. I'll play with it on Tues. Maybe we can open a SPLUS thread ?

[John]

Thanks for bringing this up, John. I overlooked SPLUS. Maybe it has exactly what I need and lack from both SAS, Matlab. glad to contribute. BTW Professor Ciresi is our in-house expert in this subject matter ;) I have learned a lot from him in the past.

[Jmuller]

I see that you folks have been discussing the relative merits of
Matlab, SPlus and SAS in this thread.

I would like to get some advice and opinions on math/stat software
for dealing with large data sets in a Windows 32 bit desktop environment.

If this is not the site or thread for such a discussion forgive me.
Maybe you can suggest where I should post.

If you are open to it … great!

Let’s say “large data set” is anything too large to fit into RAM.
i.e., maybe it fits into virtual memory but a regression takes a long, long
time due to OS paging.

Note: we are also considering solutions beyond the “Windows 32 bit desktop”.
such as move to 64-bit and some kind of server-based solution, but
for now we want to see what we can get without either of those moves.

I give my initial impressions from our search below.
Would love to hear if others have advice and experience on this topic.

best regards,
- john muller

R (Note: I have been using R for years and love it.)
R seems to have some attempts at dealing with large data sets,
e.g. SQLiteDf, filehash, g.data
I not really tried any of these yet.

Matlab. (Note: most of the folks at my office are Matlab users)
With Matlab it seems there are 2 main options: move to 64 bit or
use the DCT (Distributed Computing Toolbox).

Star-P. This is a Unix server-based distributed computing solution so
in a sense it competes with Matlab’s DCT. Seems interesting,
and especially so for me since they claim they will also support R
sometime next year. We are considering this as a server-based solution.

SAS. Not sure if the desktop version of SAS can do anything special for us
but if and when we consider server solutions this seems like a standard
choice. I don’t particularly like the SAS language, but if it meets our needs
I can live with it.

SPlus. On the surface, these folks have created exactly what we are looking for,
the bdFrame (big data frame for S and R users).
The bdFrame is stored on disk and processed in blocks, so effectively they
are doing the paging rather than the OS. You cannot use all the modeling
functions with bdFrames but you can do the most common things
(generalized linear models, clustering, principal components)
but that might be a deal breaker for some folks.
Not sure if the FinMetrics or NuOpt packages support bdFrames now
and if not if there is a plan for that.
I am experimenting with this now and so far it seems good.
It’s not rocket science, but it saves me from managing all this
writing to disk myself.

[Andy]

Let’s say “large data set” is anything too large to fit into RAM.
i.e., maybe it fits into virtual memory but a regression takes a long, long time due to OS paging.
Our large data set would be in the size of close to 50GB each so using RAM is out of the question. I don't know which of the packages will be better fit for doing number crunching on the remote data. Most software will use ODBC driver to connect and ODBC is one of the slowest.

I haven't tried SPLUS yet but it seems promising. Whatever package we use, a seamless integration with Excel is a must.

To take full advantage, it seems you need a dedicated server that runs SAS or SPLUS.
Would love to read your stories about setting up SPLUS. Sure will have questions.

[kean]

I know SAS remote call function using ODBC will work for 50GB but I don't know about S-PLUS. Why don't you call Insightful to find out? I personally like SPlus more than SAS in terms of its statistical analysis function.

Both softwares have no problem integrating with Excel.


Our large data set would be in the size of close to 50GB each so using RAM is out of the question. I don't know which of the packages will be better fit for doing number crunching on the remote data. Most software will use ODBC driver to connect and ODBC is one of the slowest.

I haven't tried SPLUS yet but it seems promising. Whatever package we use, a seamless integration with Excel is a must.

To take full advantage, it seems you need a dedicated server that runs SAS or SPLUS.
Would love to read your stories about setting up SPLUS. Sure will have questions.

[lindsay]

In my experience, in both undergraduate and graduate school, it appears that 99% of us have used MATLAB almost exclusively over anything else. But I'm from applied maths, so that may explain why.