Research to Predict NPTE Success

[RSwanson]

So, in my quest to become a better speaker, I have decided to emulate every aspect of Ankur's personage. So, I am doing some econometrics research to use things like Speaker Points, z-score, opp wins, prelim wins, etc. to model elim round success, most likely at the NPTE. As such, I need tournament data to run through tests to get a model...

So I am calling upon all you TD's and other knowledgeable people who have data, preferably in Excel format (although if you just have the data files that TRPC or some other program produces, I'll gladly take that). It would take me forever (probably to the point of me abandoning this project) to enter in all the data myself, so anything you have is greatly appreciated. Please email me at rswanson@pugetsound.edu if you can help out.

Also, if anyone has any suggestions about other predictors (independent variables) I could look into/things they would be interested in seeing, feel free to let me know via email or a post here. My exact research and method isn't set completely in stone, so if anyone has a good idea, I might incorporate it in my research.

Thanks a lot,
Rob Swanson (nerdier half of the cheetahswan)

[richmindseed]

First, I strongly suggest you get a better role model. Might I suggest the guy who outspoke everyone at your last tournament by 3 HL points? =)

Second, I would suggest the transitive W/L method created at (http://art-of-logic.blogspot.com/200...ransitive.html) as something worth looking at. If you're as nerdy as I am when it comes to the whole "how we should tab tournaments" thing, some of the older posts on strength of schedule pairings, etc might be interesting. The downside here, of course, is that computing these metrics is non-trivial (incidence matrix = sadness), but it's still an interesting look. I also suggest reading the "a numbers game" blog; they've got a lot of similar research (directed at policy debate, but still) that might give you ideas.

Third, make sure to include both JVAR and Z-Score: there is a slight difference in their calculation, so they're actually independent. If I remember correctly, Z-Score adjusts for the population mean/stddev and JVAR does not (link to an old thread explaining the precise formula was posted on the nmh thread). If I am incorrect and they BOTH adjust for the population mean, constructing an unadjusted version to examine would be interesting...

Fourth, a concern with measuring success at the NPTE would be that it's a little harder to measure elim round success in a double-elimination game: would you just look to how far a team got? Would ballot counts factor into it at all? Is there a strength-of-outround-schedule component (whether computed by seeding going into elimination rounds or by final finish of teams or by npte rankings before the start of the tournament) that needs to be included? I think the simplest solution here is to just accept as god-given the final rankings that the npte itself gives teams, since that ensures a stable metric and automatically breaks ties (which could otherwise screw with your model), but it bears thinking about.

If you can't tell, I'm a big fan of this idea, and look forward to reading whatever findings you get. I'd really like to see tournaments publishing more of this kind of data (mentioned a few ideas in that NMH thread), and I welcome any analysis that gives us more insight into our game works. Relatedly, let me know if you want any help for any part of this - I think the number-crunching is best done at one place, but there might be some other hand I could lend, so...

[RSwanson]

Ankur, that post was tremendously helpful. Thanks, and I'm sure we will be in contact (more than usual) as I go through this research.

Transitive wins looks like a nightmare to compute, especially on the scale that I am planning (I want lots and lots of data).

I think your distinction between z-score and JVAR is correct (at least my intuition says you're right). I will obviously be sure of this before I'm done...

Yes, the final rankings at the NPTE would probably be easiest.. I haven't thought any of the details through quite yet, but one downside would be that there would probably be some multicollinearity involved (seeing as how ties are broken based on prelim seed, which is determined by a number of the factors I intend to include as independent variables).

(To anyone, although especially Ankur): Feel free to suggest other independent variables for me to look into... I've gotten suggestions of number of regular season rounds, maybe total prelim win % during the regular season... all these are interesting.

[richmindseed]

Another parameter that comes to mind: #splits vs #sweeps in NPTE prelims. My gut is that teams that can win convincingly are more likely to do better, but I see an argument for "we can hang with anyone, so all we had to do was get hot in outrounds"...

Total prelim win% has the nice feature of already being computed for you via NPTE rankings. #regular season wins would have to be hand-calculated to account for where each tournament breaks to, etc...this is not hard, given that the data is on the website, but still something to do.

[asmitty]

rob--this would be difficult to compute, but one or more of the following 4 stats:

a) total prelim win % vs teams qualifying to the NPTE
b) total elim win % vs teams qualifying to the NPTE
c) total prelim win % vs teams in elimination rounds at the NPTE
d) total elim win % vs teams in elimination rounds at the NPTE

seems like it would have pretty strong predictive power.

[RSwanson]

Originally Posted by asmitty

rob--this would be difficult to compute, but one or more of the following 4 stats:

a) total prelim win % vs teams qualifying to the NPTE
b) total elim win % vs teams qualifying to the NPTE
c) total prelim win % vs teams in elimination rounds at the NPTE
d) total elim win % vs teams in elimination rounds at the NPTE

seems like it would have pretty strong predictive power.

Yeah unfortunately this would be nearly impossible since I won't have tournament data from all regular season tournaments that go into the NPTE rankings. You are probably right that this would be a pretty strong predictor.

[asmitty]

You might be able to limit your data set to all tournaments getting the full bonus--this would make the project much more manageable in scope and would probably not interfere with the validity of your findings vis a vis npte success b/c these tournaments make up the core of basically any npte elims team's schedule.

[dseltzer]

There was some discussion a couple of years ago about correlations between success at particular tournaments and placements at NPTE. Oddly enough, I think I remember something about UPS and Point Loma seeming to be predictive??? (That combination just seems so odd, but I really think someone looked at it in 2006 or so.)

[JWill]

i think elim losses should be evaluated based on ballots

i.e. if 2 teams are getting to the same elim rounds but one team is consistently dropping on 2-1's and the other is dropping on 3-0's

and that raises another issue: if they lose to a "good" team (if that's weighted into it), ballots should count; i.e. losing on a 2-1 to a good team should still show some kind of success because obviously it wasn't a crush

i think that should probably mean something . . .especially at a tournament with split panels


obviously figuring all this out would be pretty ridiculous though . . .

[aec]

You'd probably be better off spending the time researching or practicing. Not trying to discourage quantitative meta-research, which would be incredibly interesting and potentially helpful, just saying.

[RSwanson]

Originally Posted by aec

You'd probably be better off spending the time researching or practicing. Not trying to discourage quantitative meta-research, which would be incredibly interesting and potentially helpful, just saying.

This is in no way an attempt to make myself do better at the NPTE... just some research I'm doing.

[RSwanson]

so i've been thinking about this, and there's likely to be a TON of multicollinearity with a lot of these variables (total speaks and high-low speaks are probably the most obvious example, although i might run two different regressions, one with each of these two...). this means that it will probably be difficult to say for a lot of the variables which individual ones are best explaining how well teams do. any ideas for solving this?

this also means that i need lots of data to help distinguish the variables with multicollinearity. so, if your name is not joe gantt or or konrad hack (who have both very generously volunteered data) and you have tournament data, then I would be greatly appreciative if you could contact me (rswanson@ups.edu)

[RSwanson]

Can some knowledgeable person define z-score and judge variance for me and explain the difference?

thanks again for all the help,
rob

[richmindseed]

Originally Posted by RSwanson

Can some knowledgeable person define z-score and judge variance for me and explain the difference?

thanks again for all the help,
rob

Rob,

I'm not the kind of person you reference in your post, but I can try!

JVAR is, according to http://www.net-benefits.net/showpost...7&postcount=40,

JVAR = (((Assigned Score – Judge’s Mean)/Judge’s STD) * Pop STD) + Pop Mean

The metric I'm calling Z-score is simply

If
I is an index variable used for judges

and SCORE_I = (Assigned_Score - I_Mean)/I_STD

and J ={I | I judged competitor X}

Then

the Z-Score for X is SUM(SCORE_I) over J.

In other words, compute the Z-score for each person relative to their judge, and then sum over all judges the person has. This metric lacks the adjustment for population mean that JVAR does, which is the key differentiating factor. I could be wrong about this in terms of the labeling used by actual tab software (which is probably a Derek/Joe question)...calculating this from the raw data is reasonably trivial, however, so you should be okay.

[syphos]

Originally Posted by RSwanson

Can some knowledgeable person define z-score and judge variance for me and explain the difference?

thanks again for all the help,
rob

Some quick thoughts:

What's your unit of analysis? What's your dependent variable? It's difficult to suggest exact covariates if I don't know what you are doing. If the unit of observation is the team, then I think you could pilfer much of the data from the NPTE website and reshape it as necessary. My first stab at this would probably be to take placement at NPTE as the dependent variable (1-whatever place it is now), and just run a simple linear regression on it. While the data is ordinal, the dependent variable has a pretty large range that would make ordinal probit/logit useful for interpretation - also, adopting linear models seems to be a norm (at least in international relations) when the ordinal values reach eight or more categories. While it means you are violating a bunch of fun assumptions, get out of bound predictions, and it may not be BLUE, it probably still is (by mere assertion). However, make sure you follow the guidelines of whatever your professor would want you to do.

If you only do the elimination rounds, you lose a lot of information as your data is either right censored (cannot place lower than an 16) or you artificially truncate the sample and ensure that it is biased. A full listing of placing would be useful. You can also expand the sample by including all teams in the NPTE database and list any non-NPTE team as the lowest rank of the NPTE tournament. This does create some cross-observation correlation that violates a few assumptions of our standard regression models, but omitting those observations is usually a bigger sin than transforming the data. If you are particularly unhappy about having a missing observation for those cases, you could always look into multiple imputations as a way to create values for that dependent variable (basically, predicted performance), but that could be beyond what you want to do (however, it is the win for missing data).

The more standard way to control for a biased sample is a Heckman selection model. This is probably the method I would employ if I was using the 1-x rank value for the dependent variable. The first stage of the equation tests NPTE attendance (a binary dependent variable that employs probit) and the second stage tests performance at the NPTE. If you can, you want the first and second stage covariate list to be different as it is easier for each equation to be identified and, perhaps, your theory can inform the models as to what variables might influence attendance but not performance. For example, if you used google maps to find the travel distance between the NPTE tournament and the attending team's school, you would include that in the first stage, but not the second.

On multicolinearity: a variable and its modified variable may not always introduce the colinearity that is bad for a regression, and you still may be fine (for example, it is normal in the IR literature to use economic indicators as well as change in those economic indicators - and often the correlation between the two is low - we even include linear and squared terms that are highly colinear). However, if they are significantly colinear, or multicolinear, you maybe be able to drop some of those based on theory (though it sounds like you are doing this for economics, and thus, theory based empirical research is less the reason to eliminate/include variables).

I would probably include a set of 3-5 binary variables for regional scores (the base region for the excluded variable could probably be the largest one - Southwest). You will likely have to input the data, but it shouldn't take too long (though, if you are using excel instead of a command base stats package, this may be more time consuming). Likewise, once you have coded the regional variable, you can make a single binary variable if the tournament was held in that region - possibly to measure a home field advantage due to the judging pool. This binary variable serves more as an interaction term with the regional scores, but could be of interest.

If you are using multiple NPTEs over the tournament's history, then a count variable for the size of the tournament can be informative (assuming that the dependent variable is an ordinal measure of placement at the npte). Obviously, if you attended the npte in 2004, you cannot place 47th, and that would inflate your placement. Or, you could just have a control for the year it had taken place (this could be just a progressive count from the first year you observe in the data).

Also, include other relevant school specific factors: school size, private/public, 4 year or 2 year (if there is enough variation), and school endowment are all probably related to either the ability to travel or the ability to perform.

[topspin8257]

Rob,

I've been engaging in a similar analysis of results from the 2009 NFA National Tournament's L.D. results. I bring this up not because I'm trying to compete, but to point out that those results are published online in PDF format and they might be a good set of sample data to help prepare and test your programming / statistics / etc. before you apply it directly to the data you're analyzing.

I'll forward that data to the address above.

[RSwanson]

Thanks Joe!

Michael: I have yet to read over your post, but I'm sure there is some incredibly useful stuff, so I thank you in advance as well.

[UberNovi]

Originally Posted by syphos

Some quick thoughts:

However, if they are significantly colinear, or multicolinear, you maybe be able to drop some of those based on theory (though it sounds like you are doing this for economics, and thus, theory based empirical research is less the reason to eliminate/include variables).

Not trying to distract from the discussion, but can you explain the last part of that sentence?

[syphos]

Originally Posted by UberNovi

Not trying to distract from the discussion, but can you explain the last part of that sentence?

There are plenty of quantitative economics studies (and research methodologies) that are more projects in data mining and finding a good predictive model (perhaps, like the project here, but the research design is not entirely clear to me yet) rather than finding a model that is seeking explanatory causes (what I am used to dealing with). The inclusion of atheoretical variables to see what sticks in the model is more likely to induce colinearity, while a theory based approach may eliminate some of those variables due to spurious relationships - some relationships are caused by antecedent factors and ought not be included.

However, beyond the seriousness direction of the point, the broad generalization was partly in jest and was inherently wrong as most generalizations are. It's more about what you are trying to do with your empirical models and what your audience finds acceptable.

[UberNovi]

I suppose I could see your point, it just runs contrary to my experience/instruction with econometrics. Speaking in terms of OLS/Empirical studies, it was taught to me that theoretical analysis should guide the selection and analysis of explanatory variables.

Collinearity in my experience is frowned upon even if it improves the overall model fit.

Atheoretical variables should be frowned upon, such data mining projects haven't been my personal experience.

Could just be my experience. I'm not intimately familiar with political quantitative research (I'm pretty sure that's what you said your background was), so the differences between econometrics and political research could be lost on me.

On the direct topic - I think the dependent variable could easily be defined by number of ballots won at the npte. I think this helps with some of the measurement problems with defining "winning" that have been discussed.

[syphos]

Originally Posted by UberNovi

I suppose I could see your point, it just runs contrary to my experience/instruction with econometrics. Speaking in terms of OLS/Empirical studies, it was taught to me that theoretical analysis should guide the selection and analysis of explanatory variables.

Collinearity in my experience is frowned upon even if it improves the overall model fit.

Atheoretical variables should be frowned upon, such data mining projects haven't been my personal experience.

Could just be my experience. I'm not intimately familiar with political quantitative research (I'm pretty sure that's what you said your background was), so the differences between econometrics and political research could be lost on me.

The partial jab at economics is not a jab at econometrics nor econometricians. Econometrics are tools employed by a variety of fields with a diverse idea of proper application - you can read the same logit regression applied in economics, psychology, political science, or bio-sciences with a different interpretation of acceptable bias, proper refinement techniques, post-estimation analysis or sampling procedures. Much of the acceptable practice in medical journals would be reject in first round reviews in an economics or political science journal (partly due to how samples are generated, we have to be more tricky with what we do).

Generally, the school of thought I was raised in suggests that theory should generally guide our variable selection when possible - so I would agree with that sentiment expressed above. However, for a predictive model of performance, causality may not be as important - it really depends on the confines of what you want to do (or what is demanded by the assignment). For example, if you include regular seasons wins as a predictor of success of the NPTE, it is not a causal variable (does winning itself generate more wins, not directly), but perhaps it is a proxy for team strength. Generally, the variables you really want are not something you can directly measure (research hours by team, number of practices, team size, effective coaching staff, time spent on net-benefits, etc.) Yet, ideally, we want to avoid complete correlations that have little to do with the actual events we are measuring, as they will eventually be wrong, even if they are generally right:

Redskins performance predict presidential elections

Height and Name length predict presidential elections


Perhaps there is a strong correlation between success at the NPTE and who wears the pinkest shirt or top.

[RSwanson]

Time to resurrect this thread. My project has changed substantially since the beginning of this thread, as I just didn't have the right data to test what I was initially saying I was looking at. But I think I have an equally (if not more interesting) project now. I just finalized my results last night, and the paper will be coming in a little under a month, so I'll talk about my results then.

I ended up looking at differences between MPJ and strikes tournaments in terms of which prelim statistics best predict elim success. The prelim stats I looked at were wins, prelim seed, z-score, total speaks, H/L speaks, and opp wins. I compared regressions for MPJ tournaments and strikes tournaments and came up with some interesting results (yes, they are different).

I figure I'd let people know what my project is now in case they have anything to say about it. I'm also curious if anyone wants to take a stab at predicting what my results were

Hope everyone is having a good post-nationals life,
Rob

[APotter]

Sorry if Dingess and I screwed them up at all.

[RSwanson]

Originally Posted by APotter

Sorry if Dingess and I screwed them up at all.

Haha if this is a reference to you guys rocking it at this year's NPTE, then the answer is no, not at all. Mainly because this year's NPTE isn't included in the data set. But one observation (aka one team at one tournament) is hardly enough to bias the entire result (except for maybe if the very bottom breaking seed at the NPDA won the tournament) since I had 280 points for strikes tournaments and 196 for MPJ tournaments. Non-breaking teams were also not included in my data set, which contained 15 different tournaments.

[SoCalian]

did you look at (or consider looking at) tournaments that use neither MPJ or strikes?

[RSwanson]

Originally Posted by SoCalian

did you look at (or consider looking at) tournaments that use neither MPJ or strikes?

No, unfortunately I did not have the data to do that. That would, however, be an interesting subject for future research.

[syphos]

Originally Posted by RSwanson

Time to resurrect this thread. My project has changed substantially since the beginning of this thread, as I just didn't have the right data to test what I was initially saying I was looking at. But I think I have an equally (if not more interesting) project now. I just finalized my results last night, and the paper will be coming in a little under a month, so I'll talk about my results then.

I ended up looking at differences between MPJ and strikes tournaments in terms of which prelim statistics best predict elim success. The prelim stats I looked at were wins, prelim seed, z-score, total speaks, H/L speaks, and opp wins. I compared regressions for MPJ tournaments and strikes tournaments and came up with some interesting results (yes, they are different).

I figure I'd let people know what my project is now in case they have anything to say about it. I'm also curious if anyone wants to take a stab at predicting what my results were

Hope everyone is having a good post-nationals life,
Rob

I would be interested in the paper/data when you are done.

I am guessing that by comparing the two tournaments, you initially ran a pooled sample estimation followed by two identical regressions for each sub-sample?

[RSwanson]

Originally Posted by syphos

I would be interested in the paper/data when you are done.

I am guessing that by comparing the two tournaments, you initially ran a pooled sample estimation followed by two identical regressions for each sub-sample?

I first ran a pooled sample estimation, but I haven't included those results in my analysis. I compared the results from each sample directly. What additional info do you think the pooled estimate would provide?

[DEADMONEY]

^----------------Nerd

[syphos]

Originally Posted by RSwanson

I first ran a pooled sample estimation, but I haven't included those results in my analysis. I compared the results from each sample directly. What additional info do you think the pooled estimate would provide?

It depends on what your research question is. If your question is merely "what determines outround success" then I think a pooled sample is warranted as you are looking for general correlates in the population of outrounds (give your sample). However, if you just want to compare the two judging systems, then it may be less relevant.

The pooled sample is often used to see if the covariates perform well across the board in a statistically discernible manner. Generally, since you constrain the variance of the subgroups to be identical while also averaging out any of the covariates, you provide a stricter test for the hypotheses you are testing. Perhaps opp wins will always do well (strength of schedule shows promise for out round performance no matter what the format is) and that is pretty useful to know. Our regressions are looking at mean behavior and a mean of means for the regressors can inform us as to what are generally useful predictors of success without having to engage in a mpj v. strikes analysis. Also, the pool might have a better grasp at the covariates depending on the regression technique and the sample size of each subgroup - if either sample is too small (relative to the regression type), your standard errors may be growing too large to make the analysis useful.

Running the same model in two samples tells you that the two samples may be different, but you are harder pressed to treat them as proper experiments and discern causation given the selection bias and non-randomness of it all. The change in correlation across the two models are statistically hard to compare if they are not derived from the same sample and analysis risks some inferential fallacies that the statistical assumptions would not normally support. The pooled model allows you to identify variables that perform well across both samples. Generally, if something does not perform well, but it does in one particular disaggregation, you can infer some usefulness out of it.

For example, if the pooled sample and the MPJ sample provides statistically insignificant results for H/L speaks while the strikes model has significance for the same variable, I think you can devote at least a paragraph as to why that may be the case and is interesting. Deductively speaking, the argument would be straightforward, but the statistical evidence could be compelling and only really available with the pooled model.

Finally, if the pooled model provides scant information about the independent variables, but your sub-sample regressions provide strong results, that is telling as well. It could indicate that different processes are affecting these two samples and to pool them is incorrect (which we need the pooled results to know). The variables could have opposing directions and wash out significance in pooled models.

Do you have any reason to believe that the slopes for the covariates should be different for the mpj v. strike samples? If not, then you could just add a binary variable for mpj in the pooled equation as an expectation of the mean to change (teams do better/worse given mpj). However, if you think mpj or strikes mollify/enhance just 1-2 variables, then that would justify a pooled model with a binary variable for mpj as well as interactions terms for the affected slopes.