Benchmarking Notes

09 January 2003
Scribe: Jennifer Beckham

Dave Maier Presentation

  1. Some of the things that need to be addressed by a benchmark
    1. Application needs to be something streamish
    2. Establish Credibility
      1. Scale
      2. Need larger input rate and output rate
    3. Realistic: Something that cycles over a longer period
    4. Approximation
      1. Need to capture aspects of QoS
      2. Most need of a metric
    5. Expressibility is challenging: Query types and semantics may vary
    6. Portability
      1. Should not be associated with any particular data type at this time
      2. it could be implemented in a tuple-type system or XML system
    7. How should it run
      1. Can't run forever need to run once
      2. Preferably over 2 hours or at most a half day
      3. Every knob or variable that gets introduced adds time to run the benchmark
      4. Need to keep a limit on variations on similar cases
      5. There should be no way a system can win by chance (via random variables)
      6. Quality metrics must have a test harness
        1. Need an ideal answer to compare against
        2. Can measure a subset of the output, but be careful and make it hard to cheat
  2. An OGI streaming benchmark: NEXMark
    1. An eBay-type application with sellers and bidders
    2. Doesn't have high data rates
    3. Contains some static data
    4. Auction Monitoring: Bid streams scale
    5. Queries:
      1. Find Hot Items
      2. Expressed in CQL
    6. Metrics: Difference between ideal and actual is the error
    7. Scaling

Discussion

Other possibilities that need considering

Stonebraker Presentation

  1. Why have a benchmark?
    1. Gain credibility and legitimacy
    2. Need to beat the competition by an order of magnitude
    3. Get a persuasive application to beat the competition (run on Oracle and compare)
  2. Possible things to exploit in a streaming benchmark
    1. The scalable trigger system
      1. Oracle has poor triggering mechanisms
      2. Problem: Oracle may be fixing this problem (CIDR paper)
      3. it is not clear that we are noticeably better
    2. Extreme Dynamism of Queries and/or Data Rates: No real application
    3. Insert Rate
      1. Reports say that this may be fixed by Oracle for retrievals
      2. This can all be fixed in a traditional system through User Defined Functions (UDFs) or through other hacks
    4. Push processing into sensor net
      1. This is worth at least an order of magnitude over traditional systems
      2. However, this is a benchmark for TinyDB. It is not a server benchmark and Oracle can?t do it anyway.
    5. Time Series
      1. This may be worth it
        1. Incorporates sliding windows
        2. Informix/Illustra had moving average over last so many tuples and it was an order of magnitude faster than Oracle
      2. Cautionary Note: The way UDF time series work (???)
      3. If we have rich enough languages, we can eliminate boundary crossings where traditional systems have to cross multiple subsystems
    6. Exploit ACID properties
      1. QoS, dropping tuples, etc.
      2. Can you fake QoS even when you're not in a streaming environment?
      3. Aside:
        How can application servers exploit rich languages?
        Franklin: Can't express w/Oracle queries alone
        Stonebraker: We have no boundaries in streams
        Is this too hard?
        Franklin: Sharing queries is something we can take advantage of, but Oracle doesn't have this
        Stonebraker: Multi-query optimization is not done in commercial systems because every warehouse is historical streams and no warehouse user wanted multi-query optimization. Common sub expression elimination is not worth it. The important thing is to take advantage of how traditional systems have to cross multiple boundaries. An example is the work done at BEA on trying to eliminate those boundaries.
  3. Example applications
    1. Medical monitoring that is similar to Aurora application
      1. This problem is not hard
      2. W.r.t. QoS we don't want to throw away any data
    2. Military application: Mitre and Joe DiLiberti
      1. Exploits all unfair advantages
      2. Do we need to worry about being PC?
      3. We can make up the data
    3. Traffic monitoring
      1. Candidate for more thought
      2. Here it is OK to drop data
      3. There is more than one query: detect accidents
    4. Fraud detection: candidate for more thought

Discussion:

Shasha: FinTime is a time-series financial analysis benchmark
Stonebraker: The troubles are that Informix has time series extension and it will be in DB2 soon, Wall Street can't drop tuples, and financial time series can be optimized.

Summary:

Steams Unfair Advantages over Oracle

  • Can handle Time-series
  • Don't have multiple subsystems: Application + Rules + Database
  • Non-ACID qualities
  • Sharing of queries (?)
  • Widom's restatement of Unfair Advantages

  • Time series means Windows
  • Application + Rules + Database means that they have to hook together three systems
  • Non-ACID qualities means they don't have QoS guarantees, ability to handle inaccuracies, and resource shedding
  • Candidate Domains:

    Caution: Need to avoid open ended mining and need a domain expert.

    Task assignments:

  • Shasha: Look at financial time series
  • Stonebraker and Harry: Traffic and air traffic control
  • Harry-Stonebraker-Johnson: Narrow Network application
  • To Do List (3 month deadline goal) :

  • Find a Domain
  • Schema
  • Data Generator
  • Implementation in major streaming systems: OGI, Stanford, Berkeley, Brown
  • Faithful implementation in Oracle