Friday, June 26, 2015

Hack The Cause

Author:    Sean Moriarty
Project:    http:/


    A few weeks ago I was asked to start working on a project involving security log analysis via machine learning.  I had already received a bit of experience with log streaming and analysis by running a HIDS system at home but, using machine learning for analysis was new to me.  I am aware this is something that has been done before but my long term goal is to optimize algorithms specifically for threat detection and mitigation.  Again I am aware this is nothing new, I am just hoping I can do it better without insanely expensive hardware, on an open source platform.
    The first problem that needed to be tackled before this project can really start is getting data.  I needed some form of log data that I can begin to model and write my algorithms around.  I personally find it very tough to write machine learning algorithms with no data to feed to them. I decided to use Apache Flume for streaming my log data.  If you are interested in how flume works take a look at my post about setting up flue to stream log data.  The first prototype version was a static page with the text "I'm a teapot" displayed on the page.  I then wrote a quick python script to send requests to the server in random time intervals.
    This setup was great for prototyping my environment.  It gave me a pipeline to test my flume configuration and ensure that for each hit on this site the log entry was sent to my Hadoop cluster and written to HBase for later retrieval and analysis.  This also provided for a great chance to load test the pipeline and ensure flume was going to be reliable.  Once we start talking about writing machine learning algorithms this data will not provide us with very useful information.  If we want to write algorithms that will catch real bad guys we need some real data.  Preferably a server with a decent amount of normal daily traffic and some attempted breaches.  Solving the problem of getting data of this nature was a little more fun.

The Coding Begins   

    After taking a nice walk and thinking about solutions to my problem I decided I wanted to write a CTF, open it up to the world and encourage everyone to tinker away.  I had been meaning to teach myself code igniter as in my past I have heavily used Zend and I felt it was time to try something new and see how I liked it.  After a sleepless night of coding and scanning through the code igniter documentation I had a working site.  The very first version released with four beginner levels.  I bought the domain, spun up a web server  and did a soft release announcing the site to a handful of friends who are into cyber security.  The soft launch went very well without either end of my pipeline crashing over the few days.  I announced the CTF publicly and continued to put an hour in here and there as I had free time over the next week or so (I wish I could spend more time on it but work is pulling me a million different ways).  I added a few more levels a night and improved a few of the existing levels based on user feedback.  The site now has nine practice challenges with more to come as soon as I get some free time.
    On a side note I would like to thank the first handful of people who helped test and gave feedback.  The first few days the site was live it had a few really lame levels.  The SQLi stuff was all simulated using regex but, I quickly realized how silly that was and changed to a real live database being exploited.  Another note is that I understand that the levels can be played in any order by just hand typing the URL in.  This was intentional I want people to be able to skip around and play whatever levels they think are most challenging and fun for there personal style and interests.  I also realize that you can just use the JS console to call the function that displays the message for capturing a flag.  I will fix this eventually I just have not had the time and it is low priority in my mind.  You get out of a site like this what you put into it.  If you just want to call the function nine times and call it a day that is your m.o. and I won't try to stop you.  I really do appreciate the feedback, good or bad it is amazing that people care enough to send me there two cents.  I have found most all of the feedback has been positive and the few super negative responses I got were wither someone expecting way to much out of a site that has under 30 hours worth of work put into it or not understanding a level and getting frustrated.  Even that type of negative feedback provides good insight for me, I usually go back and change hints or try to make things a little but more obvious.  To a point that is, I don't want every level to be super obvious.  Even with the hints.

The Results

    The results of this project have been much better than I could have anticipated.  Big shout out to the reddit community for spreading the word and getting involved.  I myself am an avid redditor and it is really nice to be able to post something that others in the community try out and let me know how they feel about.  As of writing this the site has been up just under two weeks.  The site at this moment has just shy of 20,000 views from 4,000 unique visitors.  Between this and my internal testing that was done before I had google analytics setup we have just over a million entries sitting in HBase all from real people generating real traffic.  Over the past day I have been modeling my data to prepare it for machine learning.  I will write a part two to the first post explaining that process so keep your eyes open.

Final Notes

    Again I cant stress enough how grateful I am for every use that tried out the site.  Keep your eyes open for updates I usually add at least a new level every few days.  They are starting to take a bit longer on the coding end as they get more complex but I will try my best to put works stuff aside and spend some time on the site.  Feel free to contact me with any feedback or questions.  Thanks again and as always, happy hacking.

Sunday, June 21, 2015

Security Log Analysis And Machine Learning On Hadoop(Part 1 - Ingestion)

Author: Sean Moriarty

 For my most recent venture I have been working on using the hadoop ecosystem to setup a sandbox for analyzing log data. I chose to use hadoop for log storage for a few reasons. The most dominant of these reasons being that hadoop is what I use for work so I grab any excuse I can to get another install under my belt. On top of this hadoop provides a great set of tools for streaming, storing and analyzing log data. It also provides cheap easy scalability and storage. On the off chance that I generate terabytes of log data I can easily expand my storage capacity by adding a new datanode to my cluster.
  For my environment I went with the hortonworks stack on live and a raw hadoop install for my development environment.  I chose hortonworks for a few reasons. With my work I have had to install and manage hortonworks, mapr and cloudera stacks. I find mapr to be by far the fastest stack but due to it being much more resource intensive than its competitors I felt that was not the right way to go. I do like cloudera a lot but I tend to prefer ambari over CDH and I can't help but love supporting hortonworks due to their heavy dedication to open source, maintaining high quality code and assembling a great team of engineers.
  For ingestion of the log data I use flume. Flume is a great Apache project for real time data ingestion. It runs as agents you setup on one or more machines. Each agent consists of a source, channel and sink.  The source is just as it sounds, your data source. The channel is a passive store that keeps the event until it is sent to your sink. The sink is your datas final resting place within the agent. This may be writing your data to HDFS or passing the data to another agent via avro.  For a more detailed description of flume go here:

  In my setup I have two agents running. My first agent sits on the web server and listens for new entries in my logs.  This then sends the data to an avro sink which passes it off to an agent running on my hadoop cluster.  The agent on the cluster is configured with an avro source and an hbase sink. This means when the agent receives data into its avro source from the web server's agent it will write the entry to hbase for long term storage.  I also write the data into HDFS as a second archive should anything ever happen to hbase.  Below is the configuration file for the agent sitting on the web server.

  The above is a basic script that tails a log file and listens for new entries.  When a new entry is detected it is passed into memory as that is the channel type we have specified.  The data is then sent out over our avro sink and cleared from memory.  On the hadoop side our agent is configured to listen for the avro event being passed, write the data passed over the wire via avro into memory and send the data to our sinks.  In this configuration I have two sinks.  One writing the data onto HDFS as a flat file and one writing the data to hbase.  The configuration for this end is shown below.

  In a production environment you would want to add one more step to your agent that dumps data into hbase. This step is serializing your data. In the interest of speed I have broken my data up in python. The correct means of data ingestion would be writing a serializer class in java that will split log entries before inserting into hbase and place all data in the proper columns. Ingesting your data this way saves you from having to break the data up anytime you want to use it. For now I have written a quick python script that extracts all the data from my log entry and writes the entries to hive properly split up.  This gives me a static schema making it much easier to access my data. I then connect to hive for creating visualizations and querying data to pass off to a machine learning algorithm.
  With a working pipeline for streaming my logs it was now time to generate some data. I decided to throw together a small CTF style game and allow the world to play.  Right now it has six levels ranging from easy to moderate difficulty. This method turned out to work very well generating a little over 1.2 million log entries in the first 4 days.  The site can be found here:

  Check out the site and give the challenges a try to have your log entries added to the data pool.  In part two I will discuss modeling my data to maximize the quality of information we can get from it.  I will be talking about taking our modeled data and running it through a clustering algorithm in order to find patterns and more importantly anomalies in out data.  We will learn about looking at a users data in a manner that will give us a feel for who they are and what they are doing in our system.  All IP and related info will be blacked out in screenshots. All log data collected will be used internally only and will never be shared.

Saturday, March 21, 2015

Reverse Engineering Closed Source Android Apps Part 1(Smali Intro)

The Intro

  In my recent studies in reverse engineering android apps I have taken to learning the smali assembler/disassembler.  It is my opinion that smali is a powerful tool and learning its syntax and usage will open many doors for you in the world of reverse engineering Android applications.  Through scraping the internet I found it very difficult to find an article covering the process of disassembling, modify, rebuilding and signing apks.  In this post I hope to cover the basics and give anyone who is interested the knowledge needed to start tinkering with android apps.

The Setup:

  There are a few tool you will need to get you started.  Sadly I am writing this guide on a windows machine so my instructions will be based on that operating system.  If anyone needs help setting up an environment on a Linux machine, shoot me an email and I would be glad to help.

Android SDK

- In my practice lab I use the android SDK to write the apps that I would use in my testing.  I highly recommend the android developer studio.  It is based on IntelliJ and call me a fan boy all you want but, I love every IDE to come out of JetBrains. This also come with the Android debug bridge which will come in very handy if you plan to spend a lot of time tinkering with Android.


- You will need to have the Java JDK installed.  The two specific tools we will need are 'jarsigner' and 'keytool'.  These tools will be very important as a non signed apk will not install on your phone making it very hard to test your changes.  The process of generating a keystore and signing your apk will be described below.  You will also want to add <jdk_home>/bin directory to your environment variables so that you don't have to specify the entire path in terminal whenever you want to use keytool or jarsigner.
- You will also need to have the JRE installed. You more than likely already have the JDK but if not download and install it.


- Apktool is what we will use for decoding and building our apks.  The link below has very good instructions for installing.  I will go over the basic usage later on in this post.  For now follow the steps to download and install and verify it runs.  Apktool is what is going to generate our smali code for us.  We can then modify the smali code and rebuild the apk.  Apktool also allows for step by step debugging.  It is open source under the Apache license and very easy to setup and use.

Lets Get Started

  The first thing we will need to start tinkering is an apk.  If you are already an Android developer, skip this paragraph.  For my first smali experience I used a simple "Hello World" app.  This is very fast and easy to obtain all you have to do is start android studio and select File->New Project. This will bring up the new project prompt. Give your project a name, click next a few times and then hit finish.  When creating take note of the location of your project on disk.  You will need to know this location at a later point in time.

  Once you have this setup you just need to select Run->Run 'app' (Shift + F10). When prompted to choose a device you can just close this dialog. This will create your apk file in <project dir>app\build\outputs\apk (Example: C:\Smali\app\build\outputs\apk). In that directory you will see a file called 'app-debug.apk'. This starter hello world app is what we will use for part one of this series.

  Now that you have your apk we are going to use the decode feature of apktool in order to get our editable smali code.  Open up your terminal and navigate to the directory containing your apk. Issue the following command to decode your apk:

apktool d <apk name> 

Example: apktool d app-debug.apk

Your output should look similar to the picture below:

  If you go back to the directory containing your apk you will now see a new directory has been created. The name of the directory will match the name of your apk and contains our smali code. Open that folder with your favorite editor, I personally am a huge fan of sublime text but whatever your comfortable with will work. In this post I am going to tell you the location of the files you are looking for. When you are tinkering you may not know this information ahead of time, make sure to take advantage of the search all function in whatever tool you use to edit text files. I feel this is the best way to learn where everything is located. For instance in this example I know I want to change the text "Hello World" in my app to say something else. I could use my global search in sublime text to search for the string 'hello world'. This would return the name and location of the string in the files it resides.  

Now that we know that the value for our text field is located in the file:

 <apk location>/app-debug\res\values\strings.xml


You can open that File and you will see something that looks like the image below.

  Lets have a little fun with this text. Change the string 'Hello World!'  sitting in between the <string> tags in the 'strings.xml' file shown above to whatever suits your fancy. Now that we have changed that pesky message to something more fun we are going to rebuild our package. We will accomplish this just as easily as decoding.  In a terminal if you are not already in the directory containing your apk and the folder with your smali code(the same folder we were in for decoding). Now run the following command to build a new apk using your modified code:

apktool b <decoded folder name> -o <new apk name>.apk

Example: apktool b app-debug -o tinker.apk

  Inside of the directory you just ran the command from, you will now see a new apk by the name you provided after the '-o' in the previous command.  In my case tinker.apk.  This is the part where every guide I read when first learning smali would give you the impression that you are done.  This is not the case.  Android requires that all apps be signed in order to install them.  If you want to see it for yourself, you can put this apk on your phone using your USB cable or using the android debug bridge and attempt to install.  You will receive a message saying the install failed.  In order to test our changes we must sign the apk ourselves.  We can accomplish this using tools included in your JDK.  If it is not already there  you are going to want to add <JAVA_HOME>\bin to your path environment variable.  This is going to save you from having to type the full path to the tools we are about to use.  First up is keytool.  We will use the following command to generate a new private RSA key to use for signing our apk:

keytool -genkey -v -keystore <keystore name>.keystore -alias keystore_alias -keyalg RSA -keysize 2048  -validity 10000

Example: keytool -genkey -v -keystore tinker.keystore -alias tinker_key -keyalg RSA -keysize 2048  -validity 10000

  Set a password for your key and enter all of the requested information.  The output of the commands should look something like the picture below:

  You should have run this command from a terminal thats current working directory is where your apk resides. You will now have a new file by whatever name you specified in the above command in with your apk.  In my example we would now have tinker.keystore in the directory we ran our command from.  Now that we have the key created we need to use it to sign our apk.  We can accomplish this with the following task:

jarsigner -verbose -sigalg SHA1withRSA -digestalg SHA1 -keystore <keystore name>.keystore <apk name>.apk keystore_alias

Example: jarsigner -verbose -sigalg SHA1withRSA -digestalg SHA1 -keystore tinker.keystore tinker.apk tinker_key

  You will be prompted for the password you set while generating your key.  You should get an output that looks something like the image below:

  If you would like to verify the signature jarsigner provides you with this functionality as well.   The following command will verify that your apk was properly signed:

jarsigner -verify -verbose -certs my_application.apk

Example: jarsigner -verify -verbose -certs tinker.apk

  You will get a warning that your signature is from a certificate chain that is not verified.  This is fine it states this because you are not a verified certificate authority.  You should get output something along the lines of the message below:

  Now its time to see how closely you followed directions.  Plug your phone into your box and copy over your newly signed apk.  Open up your favorite Android file manager and browse to the location you copied the apk too.  Open the apk and allow it to install.  If all goes well you should get a message stating the app was successfully installed.  Open up your newly installed app and take a look at your message.

  This brings us to the end of part one.  Tomorrow I am going to work on part two and I hope to get it out by the end of the day. Now that we know how to convert out apk to smali code, in tomorrows post we will go over smali syntax and dig much deeper down the wormhole.  I will add some new functionality to our app and we will dig much deeper into Smali code modification.  I hope everyone enjoyed reading feel free to shoot me and email or comment.  Happy tinkering!

  If you want to know more about me my contact info can be found at

Monday, March 16, 2015

Tales Of The Unsecured Wireless Printer

    The story I am about to tell emphasizes the dangers of having an unsecured wireless printer, with a little fun in the process.  Before I go any further, the disclaimer.  The story I am about to tell was done in a test lab and not in a restaurant down the street from my house.  I photoshoped ever image you are about to see. I am superman.  This is a story about playfully teaching people the dangers of not password protecting your network devices.  That statement also applies to not changing your default login credentials.  Call me paranoid but I recommend changing the defaults even on devices that give you a secure random password.  As an example a local ISP near my home town began giving out devices that came with a random password and WPA key.  Someone discovered that they used the serial numbers as the seed for the password and key generator.  They then  reverse engineered the algorithm for generating the info and created a tool that would give you the password and WPA key for any gateway from this provider given the serial.  This story was just me having fun but I am sure you will see the potential for much more damage.
    Last weekend a friend and I were having lunch at a restaurant a few miles from my house.  We sat down, browsed over the menu and ordered just like a normal Saturday lunch.  After a few minutes my friend alerts me that they have an open wireless signal in the building.  I pulled out my phone and fired up my wifi.  Sure enough there were two open wireless signals.  One was the guest network the other was an HP printer.  Now I know that there is all sorts of fun you can have on a guest wireless network equipped with just an Android phone but, I was a bit more interested in the printer. I find wireless printers that have been setup by techs who have no security awareness can be a ton of fun. This is where the story begins.
    I connect to the printer and let DHCP hand me an address.  I opened up my settings and took a look at my IP.  I was given 192.168.223.x, leading me to assume the printer is on  Maybe or .100 but I generally start at .1 when looking for the address of a piece of hardware handling DHCP.  Sure enough gives me an HP admin page.  At this point we are going to need to check that we can actually change settings. Over the years I have seen many network devices have an open admin panel that requires authentication as soon as you try to modify anything or print.  I went into the settings to change the admin password and sure enough there was no password currently set.  I changed the password to 'fuckyou'(I know, I'm a man child).  In the image below you can see a screenshot I took of the network settings panel.

    Now that we have control of the printer the question is how do we let them know they need to secure the device?  We print out multiple copies of our favorite nerd image for the day, that's how!  First we must send them another hint though.  Changing the SSID!  In the image below if you take a close look you can see I changed the SSID to "pwn3d".  That's right I went there.  At this point I pushed down my laughter and kept clicking through the assorted tabs while brainstorming on how much ink I want to waste in the process of teaching this lesson.  I stumbled on a page going over the direct wireless printing settings.

  This page lead me to close my browser and head straight for the play store.  I searched HP and the top app was the HP direct print app.  I clicked install and spent a few more minutes doing burger in face things.  Once the app installed I fired it up and it picked right up on the printer.  Now all there base are already belong to us so, its time to queue up a print job.  At the very bottom of the post I have put the image I printed and the image of the print settings screen.  I decided to print ten copies of the image.   Full disclosure, the app maxed me at ten and I didn't feel like sitting around once the print job started to fire up multiple jobs so I left it at ten.  I payed my tab, fired up the print job and walked on out as soon as my print job told me it was successfully printed.

  As a final note, this was all done in good fun.  It is my hope that somebody got as good of a laugh out of this as I did and then proceeded to call and have someone come secure that printer.  They most likely didn't notice my changes.  I would say the most likely situation is someone saw the prints threw them away and moved on.  They most likely still have the password 'fuckyou' set as there printers admin password.  At least I had fun.

  If you want to know more about me or you are looking for someone with my skill set my contact info can be found at

Friday, March 13, 2015

Reverse Engineering Tinder's API

The Setup

    What would possible posses someone  to spend a Friday night trying to learn the inner workings of the Tinder API?  Me that's who.  I was trying to think of  a project involving social media and I thought it would be fun to write a bot that sits on tinder.  The bot would spam accept every person it finds periodically moving location for maximum reach.  The idea from there is that when a spammers account get hit it will match with my bots account.  The spam account will then send me an automated message with a link to whatever site they are advertising for.   My bot would monitor for incoming messages and when they came in run a Bayesian algorithm on the message and report the account if it is detected as spam.  The first part of making this a reality was to find out how to call the Tinder API.  I am sure someone out there has already documented this but, I feel like doing a little tinkering so this feels like a perfect opportunity.

Few Tools Necessary


    - fiddler is a wonderful free http proxy built for debugging. 
    - During the install you will be prompted to add the fiddler SSL cert to your CA.  Make sure you           do this and select the "Download to Desktop" option as well.  You will need to transfer that to             your phone(for SSL decryption) 


    - Open your shell and type "man curl" if your not familiar with curl.  You will want to be.

   I fired up fiddler and started intercepting traffic from my phone.  You can then watch the requests to the tinder API go out and read the responses.  Here is the lowdown on the process for making API calls from your shell.  The first thing you need to do is make the call to the Facebook API to get your authentication.  The easiest way to get this info is to log out and log back in form your phone while you are running though fiddler.   In fiddler you will see a request go out of your phone to  The response to this request is a JSON object that contains the access_token key.  It is the value associated with this key that is important to us.  
    Next we will open up a shell and use curl from here.  Now we are going to call the auth section of the Tinder API and start our session.   Our next command is as follows:

curl  -H "Content-Type: application/json" -d '{"facebook_token":"<your access_token>"}'

    Note the addition of the content-type header.  You will need this for your commands to properly  be interpreted by the server.  This command will return you a JSON object with the key "token".  This token is going to be very important as it is what allows us to make authenticated calls to the API.  Anytime you see me refer to auth_token this is the token I am referring too.  From here on out we are going to be adding another header to our curl calls.  We will need to add the header for our authentication,  You will see this in the next example.  From here it was just a matter of playing with anything that made an API call on my phone and looking at the URL for the request and the object that was sent and mimicking this is curl with my authentication header.  I will now give a few examples of some of the API calls.  I encourage everyone to setup a lab and play with this yourself.  You can run the same process against any API.  Throw data at the functions and see what happens.  Happy tinkering.

  If you want to know more about me or you are looking for someone with my skill set my contact info can be found at

Pull info on 20 people in your area:
curl -H "Content-Type: application/json" -H "X-Auth-Token: <your auth_token>" -d '{"limit": 20}'

Change Your Current Location:
curl -H "Content-Type: application/json" -H "X-Auth-Token:<your auth_token>" -d '{"lon":12.8067812,"lat":69.0881643}'

Like A User
curl  -H "X-Auth-Token:<your auth_token>"<user's_id>

Pass On A User
curl -H "X-Auth-Token:<your auth_token>"<user's_id>

Pull New Activity(Matches Messages Etc)
curl -H "Content-Type: application/json" -H "X-Auth-Token: <your auth_token>" -d '{"last_activity_date":"<Last Date To Check From>"}'

*Last Date is in form "2015-03-14T03:48:29.002Z"

Pull Users Info
curl -H "X-Auth-Token:<your auth_token>"<user's_id>