Benchmarking Notes
09 January 2003
Scribe: Jennifer Beckham
Dave Maier Presentation
- Some of the things that need to be addressed by a benchmark
- Application needs to be something streamish
- Establish Credibility
- Scale
- Need larger input rate and output rate
- Realistic: Something that cycles over a longer period
- Approximation
- Need to capture aspects of QoS
- Most need of a metric
- Expressibility is challenging: Query types and semantics may vary
- Portability
- Should not be associated with any particular data type at this time
- it could be implemented in a tuple-type system or XML system
- How should it run
- Can't run forever need to run once
- Preferably over 2 hours or at most a half day
- Every knob or variable that gets introduced adds time to run the benchmark
- Need to keep a limit on variations on similar cases
- There should be no way a system can win by chance (via random variables)
- Quality metrics must have a test harness
- Need an ideal answer to compare against
- Can measure a subset of the output, but be careful and make it hard to cheat
- An OGI streaming benchmark: NEXMark
- An eBay-type application with sellers and bidders
- Doesn't have high data rates
- Contains some static data
- Auction Monitoring: Bid streams scale
- Queries:
- Find Hot Items
- Expressed in CQL
- Metrics: Difference between ideal and actual is the error
- Scaling
Discussion
Other possibilities that need considering
- Fault tolerance and maintaining 24x7 uptime
- Burstiness and variability?
- What application should be used?
- Points out important characteristics of a benchmark
- Gives goals
Stonebraker Presentation
- Why have a benchmark?
- Gain credibility and legitimacy
- Need to beat the competition by an order of magnitude
- Get a persuasive application to beat the competition (run on Oracle and compare)
- Possible things to exploit in a streaming benchmark
- The scalable trigger system
- Oracle has poor triggering mechanisms
- Problem: Oracle may be fixing this problem (CIDR paper)
- it is not clear that we are noticeably better
- Extreme Dynamism of Queries and/or Data Rates: No real application
- Insert Rate
- Reports say that this may be fixed by Oracle for retrievals
- This can all be fixed in a traditional system through User Defined Functions (UDFs) or through other hacks
- Push processing into sensor net
- This is worth at least an order of magnitude over traditional systems
- However, this is a benchmark for TinyDB. It is not a server benchmark and Oracle can?t do it anyway.
- Time Series
- This may be worth it
- Incorporates sliding windows
- Informix/Illustra had moving average over last so many tuples and it was an order of magnitude faster than Oracle
- Cautionary Note: The way UDF time series work (???)
- If we have rich enough languages, we can eliminate boundary crossings where traditional systems have to cross multiple subsystems
- Exploit ACID properties
- QoS, dropping tuples, etc.
- Can you fake QoS even when you're not in a streaming environment?
- Aside:
How can application servers exploit rich languages?
Franklin: Can't express w/Oracle queries alone
Stonebraker: We have no boundaries in streams
Is this too hard?
Franklin: Sharing queries is something we can take advantage of, but Oracle doesn't have this
Stonebraker: Multi-query optimization is not done in commercial systems because every warehouse is
historical streams and no warehouse user wanted multi-query optimization. Common sub expression
elimination is not worth it. The important thing is to take advantage of how traditional systems have to
cross multiple boundaries. An example is the work done at BEA on trying to eliminate those boundaries.
- Example applications
- Medical monitoring that is similar to Aurora application
- This problem is not hard
- W.r.t. QoS we don't want to throw away any data
- Military application: Mitre and Joe DiLiberti
- Exploits all unfair advantages
- Do we need to worry about being PC?
- We can make up the data
- Traffic monitoring
- Candidate for more thought
- Here it is OK to drop data
- There is more than one query: detect accidents
- Fraud detection: candidate for more thought
Discussion:
Shasha: FinTime is a time-series financial analysis benchmark
Stonebraker: The troubles are that Informix has time series extension and it will be in DB2 soon,
Wall Street can't drop tuples, and financial time series can be optimized.
Summary:
Steams Unfair Advantages over Oracle
Can handle Time-series
Don't have multiple subsystems: Application + Rules + Database
Non-ACID qualities
Sharing of queries (?)
Widom's restatement of Unfair Advantages
Time series means Windows
Application + Rules + Database means that they have to hook together three systems
Non-ACID qualities means they don't have QoS guarantees, ability to handle inaccuracies, and resource shedding
Candidate Domains:
- FinTime
- Automobile Traffic monitoring
- Parameters of queries changing
- Can think of people in cars running queries (car-centric)
- Rules you can't do with database triggers
- Personalized output
- Air Traffic Control
- Each Plane has many sensors
- Can look at the impact of free flight rules instead of flight paths
- Simulation model can use old data
- Fraud
- Network Monitoring/Gaming
- Auction
- Click Stream: Troubles an overrun market and MSR may be tackling it.
- Manufacturing Plant
- US Squirrel (Air) Corps
- Intrusion detection
- Caution: if all rules can fold into one rule with parameters then it can all fit into a table and can be done in Oracle
Caution: Need to avoid open ended mining and need a domain expert.
Task assignments:
Shasha: Look at financial time series
Stonebraker and Harry: Traffic and air traffic control
Harry-Stonebraker-Johnson: Narrow Network application
To Do List (3 month deadline goal) :
Find a Domain
Schema
Data Generator
Implementation in major streaming systems: OGI, Stanford, Berkeley, Brown
Faithful implementation in Oracle