The Development, Test, Evaluation, Verification & Validation (DTEVV) Workshop

Introduction

Overview of the DTEVV Graduate Certificate and Proposed
Master's Program

Stephen Cricchi
Executive Director
Naval Air Warfare Center Aircraft Division (NAWCAD)

Introduction by Donald "Bucket" Costello III

AM Panel: "DTEVV Best Practices"

A strong influence on DTEVV Best Practices is the workforce that devises them.
Although T&E/V&V is not a new phenomenon, it has undergone considerable growth
and transformation both as its importance has become better understood and how
critical it is to the development and adoption of intelligent systems. DTEVV professionals,
from those early in their careers to seasoned practitioners, develop and continually
refine best practices that guide their efforts. This panel will explore best practices and
how their growth is influenced by an ever-evolving workforce. Some of the discussion
will center around key skillsets that have had a positive impact on best practices.

Moderator
Brian A. Weiss
Research Engineer
University of Maryland Applied Research Laboratory for Intelligence and Security (ARLIS)

Panelists
Theresa Shafer
Chief Technology Officer
NAWCAD

Additional panelists to be announced

Lunch and PM Keynote: "The Human Element: Preparing T&E Professionals for
Tomorrow's Challenges"
(Lunch will be provided)

The Test and Evaluation (T&E) community serves as the cornerstone of ensuring that
technologies critical to national security meet rigorous standards of safety, reliability,
and effectiveness. As the Department of Homeland Security (DHS) confronts increasingly
complex threats and rapidly advancing technologies, the T&E workforce must remain
grounded in the fundamentals of its mission while evolving to address emerging challenges.
This speech explores the foundational principles that must guide the T&E workforce,
emphasizing technical expertise, systems thinking, and adaptability. By reinforcing these fundamentals, the T&E community can ensure DHS remains prepared to meet the demands
of tomorrow’s security landscape. 

Wayne Dumais
Deputy Director, Office of Test and Evaluation, Immigration and Border Security Programs
U.S. Department of Homeland Security

Introduction by Brian A. Weiss

PM Panel: "DTEVV Skillset Needs"

T&E, and the overall DTEVV community, is only as effective as their workforce capability.
Some personnel enter the workforce with key T&E skills that they built through their
undergraduate/graduate education or through internships. Conversely, some practitioners
enter the workforce with adjacent skillsets where they dramatically build their T&E prowess
through ‘on-the-job’ training and ‘getting their hands dirty’ in actual T&E efforts. Neither is
better than the other - it is often that both provide complementary skills that ultimately boost
the finished product - the successful development and assessment of an emergent technology.
This panel will explore the critical DTEVV skills, either built through formal education or in the workplace, that have shaped the DTEVV community. Likewise, this panel will explore the expected skills that will be needed to develop, implement, and assess the technologies of tomorrow.

Moderator
Donald "Bucket" Costello III
Director of Test and Evaluation of Autonomous Systems
University of Maryland MATRIX Lab

Panelists
Mauricio Castillo-Effen
Fellow
Lockheed Martin Advanced Technologies Laboratories

CAPT Daniel "Tonto" Kitts, USN
Commodore, Naval Test Wing Atlantic
NAWCAD

Additional panelists to be announced

Opening Remarks

AM Keynote: "From 'Cool' to Confident: The Unavoidable Journey of Robot Testing
and Evaluation"

Robot performance evaluation is often treated as an afterthought when we are heads
down, working on something cool, and rightly so. Methodical performance evaluation
can sometimes slow down this process of discovery. Nevertheless, we eventually have
to test (formally or informally) whether the thing we conjured up can accomplish the
task it was created to complete. This talk will present a brief history of the development
of performance metrics, test methods, artifacts, and standards for robots and autonomous
systems (and their related technologies) at the National Institute of Standards and Technology (NIST), with examples (from multiple domains) of successes and failures, and the lessons learned from both. Full Abstract

Kamel S. Saidi
Program Manager and Group Leader, Intelligent Systems Division
National Institute of Standards and Technology (NIST)

Introduction by Brian A. Weiss

AM Panel: "Emergent DTEVV Research Areas"

The increasing importance of test and evaluation has bred various lines of T&E-driven
research, often occurring in parallel with specific technology development. From
robotics to autonomous systems, T&E/V&V practices have had to evolve to ensure they
capably and rigorously assess emergent technologies. This panel will dive into some of
these T&E research activities and the driving technologies along with the practical insight
that they have gained in their efforts. Panelists will be further engaged on successes, even early ones, best practices they have developed along with challenges they have faced
along the way.

Moderator
Brian A. Weiss
Research Engineer
University of Maryland Applied Research Laboratory for Intelligence and Security (ARLIS)

Panelists
George Hwang
Chief Technologist, AI/Autonomy Division, Advanced Development Department
Naval Air Warfare Center Aircraft Division (NAWCAD)

Craig Lennon
Scientist
Army Research Lab

Jeremy Marvel
Computer Scientist
NIST

Additional panelists to be announced

Speaker to be announced

Introduction by Donald "Bucket" Costello III

PM Panel: "Predictions About What is Needed in T&E"

The technological landscape is rapidly evolving, particularly with the exponential growth
of artificial intelligence (AI), machine learning (ML), and the resultant systems they are
instantiated. Even for the seasoned technologist, it was hard to predict the current state of
technological development ten years ago. Considering the current state of AI and ML-driven technologies, it will likely be more difficult to predict the technological growth, and the
corresponding T&E/V&V efforts, over the next ten years. This panel will explore the arc of technology, emphasizing the DTEVV of autonomous systems in recent years and discuss
how the landscape will continue to evolve to meet the demand of predicted technologies.

Moderator
Donald "Bucket" Costello III
Director of Test and Evaluation of Autonomous Systems
University of Maryland MATRIX Lab

Panelists
Oleg V. Sokolsky
Program Director, Computer and Information Science and Engineering (CISE)
U.S. National Science Foundation (NSF)

Additional panelists to be announced

Donald "Bucket" Costello III
Director of Test and Evaluation of Autonomous Systems
University of Maryland MATRIX Lab

Brian A. Weiss
Research Engineer
University of Maryland Applied Research Laboratory for Intelligence and Security (ARLIS)

Stephen Cricchi
Executive Director
Naval Air Warfare Center Aircraft Division (NAWCAD)

Craig Lennon
Scientist
Army Research Lab

Wayne Dumais
Deputy Director, Office of Test and Evaluation, Immigration and Border Security Programs
U.S. Department of Homeland Security

Jeremy Marvel
Computer Scientist
National Institute of Standards and Technology (NIST)

Kamel S. Saidi
Program Manager and Group Leader, Intelligent Systems Division
NIST

Extended Stay America
240-725-0100
46565 Expedition Park Drive
Lexington Park, MD 20653
5.4 mile drive from the USMSM campus

Fairfield Inn
301-863-0203
22119 Three Notch Road
Lexington Park, MD 20653
6.4 mile drive from the USMSM campus

Hampton Inn
301-863-3200
22211 Three Notch Road
Lexington Park, MD 20653
5.9 mile drive from the USMSM campus

Holiday Inn Express & Suites
240-237-1444
45260 Abell House Lane
California, MD 20619
2.6 mile drive from the USMSM campus

Home2 Suites by Hilton
301-866-1416
46058 Valley Drive
Lexington Park, MD 20653
6.3 mile drive from the USMSM campus

The Inn at Leonardtown
301-475-0187
41655 Park Avenue
Leonardtown, MD 20650
8.6 mile drive from the USMSM campus

La Quinta Inn & Suites by Wyndham
301-678-9847
22769 Three Notch Road
California, MD 20619
3.8 mile drive from the USMSM campus

TownePlace Suites
301-863-1111
22520 Three Notch Road
Lexington Park, MD 20653
4.6 mile drive from the USMSM campus