Sunday, April 5, 2015

Redis for Service Monitor Visibility

From #redis on FreeNode (made anonymous and more succinct):
Q: Is redis a good choice if I need a solution to order my result and filter it, or should I use mysql for this job?  I'm developing an interface to show Nagios status information and for this I need to order by state (up, down, unknown) or state since and so on.  I played with Memcached, then found Redis, but I'm not sure if it is a good idea to use Redis for a job like this.

An answer lead to this being implemented in MySQL instead.  So the original poster added a desire to protect MySQL from 12K inserts (without specifying the period).

12K inserts per minute is barely doable for a single MySQL server with the table being indexed for the queries specified.  However, with doubt about the period, a MySQL solution will likely need to be modeled to match the write and read patterns as well or performance will suffer in time.  So data modeling should be performed.

A: For monitoring data, as with all data in Redis, model the data as it will be accessed.  There is highly likely a tool that already provides the feature set of monitoring, likely using a timeseries data store (see Circonus instead of Nagios).  However, this seems like a good case for understanding how to model data when using Redis.

I'd imagine the following queries: (1) What host/service are in the state {up|down}?  (2) What hosts are hosting service?  (3) What is the data for a host/service, such as {host, service, os_version, service_version, state, last_monitor_time, last_up_time, last_down_time}?  (4) What are the last n status monitor messages received for a host/service?  In short, 1) is a couple sets for {all|down} with {up} being derived by diff, 2) is a set, 3) is a hash, 4) is a list of finite size left pushing + right popping on insert.

In longer form, (Using #{key_part} to indicate templated values)...
1) could be modeled as sets "mon:hostservices:all" and "mon:hostservices:down" with "mon:hostservices:up" being achieved by a set difference and the host/service being either added or removed from the "mon:hostservices:down" when a monitor event is processed.  Items in the set are in the form of the identifier used in the item's data (see 3).
2) is modeled similar to "...all" in 1 and achieved by adding to the "mon:hostservices:#{service}" when a monitoring event is processed, adding the host as the item value.  The interesting bit is how host/services are removed.  A reaper process could be employed to iterate over all host/services, removing "mon:hostservices:all" and "mon:hostservices:down" and "mon:hostservices:#{service}" set entries that have a last_monitor_time (see 3) that is expired (longer ago than considered active).
3) could be modeled as a hash "mon:hostservices:#{host}:#{service}" with the fields set straightforwardly from the monitor event, with the exception of last_up_time or last_down_time which the field is chosen based on whether the event is an up or down state indicator.
4) could be modeled as a list "mon:hostservices:#{host}:#{service}:events" with finite size, pushing the message on the left-hand side while popping the right-hand side of the set.

The above model is intentionally simplified.  Additional queries may also be desired, such as service availability should be modeled as timeseries counters for downtime.  Timeseries counters are may be implemented using zset with a partitioned timestamp as the member and the score being the counter value.  The key for the zset in this case would be "mon:hostservices:#{host}:#{service}:counters:downtime".  The downtime counter would be incremented whenever a monitoring event indicated a downtime and the increment by would be the monitoring period.  Again, a reaper may be employed to remove data that is older than desired to be retained.

While this may be a good case for understanding data modeling in Redis, the cost of development and operating an in-house developed solution is highly likely more than the cost of licensing and implementing a monitoring service that provides the desired functionality.  The benefit of modeling, though, should help in selection, so is a fruitful exercise.

Thursday, March 26, 2015

Enjoying Computing, Enjoying Erlang

Nearly 33 years ago I got the spark for computing when I saw my uncle's FORTRAN at work, simulating the flight of bees across tarmacs and the impact of such seemingly trivial things over time.  His work was wondrous to me and left a very meaningful effect on my understanding of the importance of information, where to measure it, how to process it, and all the while to work with the knowledge that the smallest of things can have great impact.

Nearly a week ago, I began evaluating Riak and am sparking again from the architecture, the support from Basho, and the elegance of Erlang.  Through functional argument matching, functions are simplified, matching on predicates rather than branching on imperative logic.  Through modules dependencies are clearly delineated.  Through actors, distributed computing is modeled in a manner that any consumer of a (e)mail system should understand.  This all comes at some performance cost, but is generally overcome by the greater performance gains of concurrency and simplified exception handling.  And, the exercise of optimizing simple recursive functions in a tail-recursive manner is a joyful experience.  Built-in profilers bring the joy of chasing code coverage and eeking out higher and higher requests per second a game for fun and profit.

For the yet to be deep into the world of imperative programming, Erlang might bring you the wonder that those digital bees brought me.  For any who have had tastes of functional programming, concurrency, and dynamic languages, Erlang may be quite enjoyable.  I found the following to be a well written, funny at times, foray into the language:

Monday, March 9, 2015

MVC from a CRC Perspective

The Model-View-Controller (MVC) pattern maps well to programming an HTTP client interface with persistent data stores as each external system interface requires a decent amount of protocol-specific adapter code which is well focused by the pattern.

While measuring the effectiveness the MVC pattern in focusing code according to component purpose, several anti-patterns have arisen, such as the following:
  • Spaghetti View
  • Fat Model
  • Fat Controller

Instead of delving into the anti-patterns, we should understand first the intent of the pattern and what force(s) cause the pattern to break down, giving rise to the anti-patterns.

The MVC pattern from a Class Responsibility Collaborator (CRC) perspective follows, specifying for a network service, listing classes according to the trace of a client call path:
  • Router
    • List the business operations of the domain
    • Route business operation requests to the appropriate Controller
    • Translate InterProcessCommunication (IPC) protocol semantics, it HTTP, into functional programming semantics.
  • Controller
    • Receive Client business operation requests
    • Validate parameters
    • [optional, should be omitted due to optimization for a postcondition-guaranteed system] Validate system state wrt the requested operation
    • Coordinate Model operations
    • Respond to Client with Resource or Error
  • Model
    • Receive CRUD operation requests
    • Translate DataStore protocol semantics into a common CRUD set of operations.
    • Respond to Client with Resource or Error
  • View
    • Receive Model
    • Translate Model into a format fit for consumption

With the MVC CRC listing in mind, developers relatively easily can create, read, update, and delete code in the appropriate class when programming a feature that exposes operations on a single resource.  This leads to quick "green field" development as well as small, incremental modifications such as exposing a display_name as a calculated field on a Person resource when the resource had only the constituent parts as fields.

From measures of deviance in primarily MVC-patterned applications, anti-patterns emerge typically as a result of development technical and feature increments that cause strain at points not addressed in the basic model of the pattern, such as:
  • multiple underlying data stores become necessary to store a resource
  • resource modifications become cohesive and through incorrect channels, ie models interact
  • deviance from model operations being restricted to CRUD, but instead expanding to be the set of business operations

A large percentage of deviances from quality MVC-patterned code are the result of basic code handling mechanics which are necessarily made more difficult when employing MVC.  For instance, when altering the constraints on a resource, the change must be performed on the model as well as on the controller (and likely on a view or a few).  This opposition to the DRY principle is evident within the perceived need that Ruby on Rails and Django code generators fulfilled.  No pattern or practice is perfect; as such, it is incumbent on the programmer to know their pattern and perform the alteration with minimal negative impact on the code through the development channel (through QA and up through delivery to the consumers of the solution.)

Other antipatterns in MVC emerge as a result of strains that are outside of the scope of the MVC pattern itself, ie multiple underlying datastores.  A general solution that often proves successful is to employ the Façade pattern, creating a ResourceModel (no datastore) that translates one-to-many underlying models into a concise, coherent, and clear solution to an otherwise jumbled solution.  The Façade pattern also resolves the strain caused when attempting to resolve how a model can be restricted to support only CRUD operations.

If you would experience MVC antipatterns in the wild, please wrangle them.  If you would like to share your experience, please comment.

Wednesday, February 18, 2015

ARP Poisoning mitigation with or without DAI

ARP poisoning, a mainstay of obtaining "man in the middle" position, persists despite advances in computing processing power and memory availability to switch manufacturers.  There are however current solutions and theoretical advances in the field which do not require expensive stateful packet analysis.

Cisco Catalyst 6500 series ( ) offers Dynamic ARP Inspection (DAI), a feature which best practice recommends to be enabled, but is not enabled by default.  DAI basically utilizes ARP table verification, putting some of the additional memory capacity and processing power to use.  The feature, however, depends upon DHCP snooping, thus upon the accuracy of DHCP data.  Despite the potential hole exposed by the dependence upon DHCP, DAI is likely to mitigate an even vaster majority of ARP poisoning attacks in the wild as the feature has mitigated in the lab.

Alternative solutions may be developed without the dependence upon DHCP snooping, but employing similar tradeoff analysis.  For example, subsequent attempts to associate a MAC with an IP Address can be limited as well as progressively throttled.

The question of how to handle when limit is reached beyond the naive "reject at limit" may call for protocol changes.  Protocol changes are, for good, practical reason, not only discouraged but a path taken at one's own peril.  The number of devices which implement ARP at the current protocol level is vast.  While the devices which we personally operate may be easily patched, patch drift is a reality due to the perceived cost/benefit of remaining up-to-date.  Nearly all networking devices utilize firmware so support patching, but patch drift of devices is larger than that of personal computers.  Therefore, the parallel support for a proposed version and current versions of the protocol have a real cost that is extremely high.  IMHO it is thus best to not employ limiting.

Progressive throttling of ARP while strengthening the certainty of a MAC to IP association would require relatively cheap (no control flow) stateful packet inspection, but would severely limit the window of opportunity and potential for ARP poisoning.  Without stateful packet inspection, progressive throttling has opportunity to succeed in preferring the actual addressee over the snooping man in the middle.

Throttling and limiting is just one example of techniques from application layer that may be employed in the networking layer.  In the meantime, DAI mitigation success should continue to be enabled as per best practices.

Monday, July 14, 2014

Naughty Beginnings

Attached is the beginnings of a game that we plan to use to test a new AI being.  This is not the most stringent test, in fact it is a children's game.  At this point, I began coding the solution, but have to run to the lab to put out a fire that the AI being started while throwing a tantrum (and a few technicians).

Get the gist:

Please complete the HumanObserver.display method for a command-line interface (CLI) or HTML interface.

Please don't get distracted by the remaining classes in the overall design.  I was simply blocking out the design ahead.  Once we have the display, I expect the project will iterate rather rapidly.

Thank you.  I look forward to seeing some naughts, crosses, and elegant code.

Background: This is a Ruby adaptation of a interview question that was so popular in the hiring of C programmers that I believe it to be influential in the making of the movie "War Games".

Tuesday, July 8, 2014

Wishing you great success

Given the following definitions of success:
1. the favorable or prosperous termination of attempts or endeavors
2. the attainment of wealth, position, or honors
3. a performance or achievement that is marked by success
4. a person or thing that has had success
5. the difference between realized and expected value
6. the correct or desired result of an attempt
7. the opposite of failure
8. going from failure to failure without loss of enthusiasm
9. the result of a desire for success being greater than the fear of failure
10. the result of rising early, working hard, and striking oil

A. Which is nearest to your definition?

B. Which is the nearest to your ideal boss?

C. Which is the nearest to your ideal colleague?

D. Which is the nearest to your ideal dependency?

E. Which is the nearest to that which you would wish upon your child?

F. What type of person would you attribute to each?  For example, given a definition of "success is the cause of more work" could be attributed to a dependable worker.

Wednesday, October 9, 2013

QA Automation a la Westinghouse

No single technological advance meant more for a maturing railroad industry than the invention of the air brake….

This like the story of the invention of the computer programming language C++ are occasions where automation not only made a hugely significant impact, but also they are stories that are relatively open, providing great insight into the minds of great automators.

George Westinghouse, like Nikola Tesla (who Westinghouse employed), did not invent from nothing or just work hard. Where Tesla had decided upon AC as the solution upon seeing how obviously poor performant the brush-based DC solutions must be, Westinghouse decided upon an engineer-driven railroad braking system similar to how a horse-drawn carriage driver pulls the reigns, Westinghouse envisioned the engineer pulling reigns of some sort to apply brakes on every car (this couldn’t be employed with a physical connection, ie multiple levers pulling a metal shaft per car). Westinghouse’s great epiphany towards solving the problem of brakemen running on top of the cars and the scaling and (life) cost and poor stopping performance of the manual solution did not come from banging his head on a train-specific problem, his great epiphany came upon hearing news of an air-driven drill employed in Italy to excavate minerals. The simplicity of the invention in hindsight is awesome. The length of time to implement less so. The end result, though, the train industry scaled, and in the expansion was able to remove the need to have men run atop the train cars, saving lives, but more importantly to the owners increasing the number of cars at the command of the engineer who was nearest the upcoming rail obstacles.

With QA being the engineers with the clearest sight of obstacles (and opportunities since we aren’t on a rail), automation is our salvation.