You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-12 - 09:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Thu Mar 12, 2026 00:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
79631999475452,20cyclictest847976-21turbostat21:39:562
79632799468447,18cyclictest849693-21turbostat21:44:563
79632799468447,18cyclictest849693-21turbostat21:44:553
79632799465456,6cyclictest893650-21turbostat23:49:553
79632799464455,6cyclictest898847-21turbostat00:00:003
79632799464455,6cyclictest808898-21turbostat19:45:003
79631999464457,4cyclictest865579-21turbostat22:29:552
79632799463454,6cyclictest886719-21turbostat23:29:543
79631999463455,5cyclictest909546-21turbostat00:34:552
79632799462453,6cyclictest913003-21turbostat00:40:013
79632799460453,4cyclictest837579-21turbostat21:04:593
79631999456447,6cyclictest855197-21turbostat21:59:562
79632799455447,5cyclictest872559-21turbostat22:45:003
79631999454446,5cyclictest902308-21turbostat00:14:542
79632799452445,4cyclictest890185-21turbostat23:39:553
79632799452445,4cyclictest881518-21turbostat23:10:003
79632799452444,5cyclictest860391-21turbostat22:14:543
79632799450442,5cyclictest900588-21turbostat00:05:003
79631999450423,24cyclictest867309-21turbostat22:34:552
79632799449442,4cyclictest863854-21turbostat22:24:553
79632799449440,6cyclictest834127-21turbostat20:55:003
79632799449440,6cyclictest823092-21turbostat20:25:013
79631999449442,4cyclictest834127-21turbostat20:59:552
79632799447440,4cyclictest842784-21turbostat21:24:553
79632799447429,15cyclictest878060-21turbostat23:04:543
79632799443434,6cyclictest883245-21turbostat23:15:003
79632799443425,5cyclictest816172-21turbostat20:05:013
79632799442435,4cyclictest904047-21turbostat00:15:013
79631999441434,4cyclictest826871-21turbostat20:39:552
79632799440432,5cyclictest837579-21turbostat21:09:553
79632799439432,4cyclictest808898-21turbostat19:49:553
79632799439432,4cyclictest808898-21turbostat19:49:553
79632799439431,5cyclictest853473-21turbostat21:54:553
79632799437430,4cyclictest884984-21turbostat23:20:003
79632799437430,4cyclictest884984-21turbostat23:19:593
79632799436412,21cyclictest821368-21turbostat20:20:003
79632799434412,19cyclictest888449-21turbostat23:34:553
79632799433426,4cyclictest797978-21turbostat19:15:003
79632799432425,4cyclictest830624-21turbostat20:49:553
79632799432423,6cyclictest884984-21turbostat23:24:543
79632799432423,6cyclictest810634-21turbostat19:54:553
79631999430422,5cyclictest805439-21turbostat19:39:562
79632799424399,21cyclictest878060-21turbostat23:00:003
79632799420413,4cyclictest803698-21turbostat19:29:593
79632799420412,5cyclictest812405-21turbostat19:59:553
79631999420411,6cyclictest904047-21turbostat00:19:562
79632799419412,4cyclictest846236-21turbostat21:34:543
79631999418410,5cyclictest819627-21turbostat20:19:542
79632799414407,4cyclictest797978-21turbostat19:19:553
79632799413403,7cyclictest828582-21turbostat20:44:553
79631399406399,4cyclictest891912-21turbostat23:44:560
79632799404397,4cyclictest817901-21turbostat20:10:013
79631999403396,4cyclictest869036-21turbostat22:39:542
79632799400391,6cyclictest858652-21turbostat22:09:563
79632799400391,6cyclictest858652-21turbostat22:09:563
79632799397390,4cyclictest801928-21turbostat19:25:003
79632799395387,5cyclictest858652-21turbostat22:05:003
79631999395388,4cyclictest883245-21turbostat23:19:552
79631999395388,4cyclictest883245-21turbostat23:19:552
79631599395361,21cyclictest870827-21turbostat22:44:551
79632799392369,20cyclictest900588-21turbostat00:09:563
79632799390382,5cyclictest909546-21turbostat00:30:003
79632799389381,5cyclictest863854-21turbostat22:20:003
79632799385377,5cyclictest805439-21turbostat19:35:013
79631399385378,4cyclictest823092-21turbostat20:29:550
79631399385378,4cyclictest823092-21turbostat20:29:550
79632799383377,4cyclictest846236-21turbostat21:30:003
79632799381375,3cyclictest911273-21turbostat00:35:003
79632799380373,4cyclictest839318-21turbostat21:14:553
79632799378371,4cyclictest835857-21turbostat21:00:003
79632799375368,4cyclictest807174-21turbostat19:40:013
79632799374365,6cyclictest907512-21turbostat00:25:003
79632799371363,5cyclictest841046-21turbostat21:19:553
79632799363357,3cyclictest828582-21turbostat20:40:003
79632799362356,3cyclictest824827-21turbostat20:34:553
79632799362356,3cyclictest824827-21turbostat20:34:553
79632799360353,4cyclictest856913-21turbostat22:00:003
79631999358351,4cyclictest817901-21turbostat20:14:562
79631999358351,4cyclictest817901-21turbostat20:14:552
79632799354345,6cyclictest849693-21turbostat21:40:003
79631999344337,4cyclictest895377-21turbostat23:54:562
79631999343336,4cyclictest821368-21turbostat20:24:552
79632799341334,4cyclictest819627-21turbostat20:14:593
79632799341334,4cyclictest819627-21turbostat20:14:593
79632799335329,3cyclictest905783-21turbostat00:20:003
79632799330323,4cyclictest870827-21turbostat22:40:013
79632799329321,5cyclictest893650-21turbostat23:45:003
79631999326317,6cyclictest851723-21turbostat21:49:542
79631399324317,4cyclictest832393-21turbostat20:54:550
79632799323314,6cyclictest874606-21turbostat22:50:003
79631399312305,4cyclictest874606-21turbostat22:54:560
79631399312305,4cyclictest874606-21turbostat22:54:550
79632799302295,4cyclictest853473-21turbostat21:50:003
79632799147134,9cyclictest824827-21turbostat20:30:013
79632799147134,9cyclictest824827-21turbostat20:30:003
79632799138125,9cyclictest897109-21turbostat23:55:003
796005213373,22sleep00-21swapper/019:07:500
90705821240,7sleep0161rcuc/000:20:270
1973991200,100rtkit-daemon0-21swapper/219:08:302
81178621180,15sleep234-21ksoftirqd/219:50:282
796075210875,23sleep30-21swapper/319:08:453
795914210775,23sleep10-21swapper/119:06:371
7963199910595,7cyclictest905783-21turbostat00:24:552
7963279910392,8cyclictest869036-21turbostat22:35:003
7963139910091,6cyclictest879795-21turbostat23:09:550
796319999992,4cyclictest803698-21turbostat19:34:542
796319999790,4cyclictest856913-21turbostat22:04:552
796327999689,4cyclictest867309-21turbostat22:30:003
796327999080,7cyclictest876331-21turbostat22:55:003
796327999080,7cyclictest876331-21turbostat22:54:593
197399820,9rtkit-daemon2212-21gvfs-afc-volume22:10:181
8307562690,7sleep2331rcuc/220:45:012
79631399550,51cyclictest1972-21rtkit-daemon20:41:370
197399530,8rtkit-daemon1972-21rtkit-daemon20:49:041
8064182510,6sleep0806420-21awk19:35:210
8479472480,16sleep126-21ksoftirqd/121:34:551
796313994824,9cyclictest13-21ksoftirqd/000:35:000
796319994721,12cyclictest141rcu_preempt19:14:502
796319994513,5cyclictest141rcu_preempt23:00:002
796319994324,15cyclictest34-21ksoftirqd/220:47:282
796315994320,11cyclictest803594-21systemd-udevd19:29:191
79631599431,14cyclictest2069-21gnome-shell20:29:081
79631599431,14cyclictest2069-21gnome-shell20:29:081
8127342420,5sleep1251rcuc/119:55:021
796315994232,6cyclictest3692981-21snapd22:52:351
796315994232,6cyclictest3692981-21snapd22:52:341
79631399421,23cyclictest831261-21diskstats20:45:150
197399420,28rtkit-daemon438-20systemd-journal19:29:172
79631999419,5cyclictest141rcu_preempt23:45:222
796319994123,6cyclictest34-21ksoftirqd/222:45:012
796319994114,5cyclictest141rcu_preempt21:25:182
79631599415,6cyclictest26-21ksoftirqd/119:30:221
796315994120,18cyclictest851416-40apt-check21:41:271
796315994120,18cyclictest851416-40apt-check21:41:261
796313994114,11cyclictest141rcu_preempt23:45:310
796313994113,10cyclictest13-21ksoftirqd/020:14:240
796313994113,10cyclictest13-21ksoftirqd/020:14:240
796313994112,12cyclictest842814-21cat21:20:010
8043332400,7chrt4497-21PK-Backend19:30:140
79631999407,9cyclictest141rcu_preempt23:10:282
79631999402,13cyclictest909550-21munin-run00:30:002
796319994014,6cyclictest141rcu_preempt22:10:012
796319994014,6cyclictest141rcu_preempt22:10:012
79631999401,11cyclictest881250-21taskset23:06:452
79631399407,19cyclictest0-21swapper/020:08:000
796313994022,14cyclictest811292-21fw_packets19:50:180
796313994022,13cyclictest13-21ksoftirqd/022:44:550
796313994013,9cyclictest13-21ksoftirqd/023:33:590
79631399401,15cyclictest890655-21taskset23:35:140
79631999396,6cyclictest141rcu_preempt23:40:022
796319993914,6cyclictest141rcu_preempt22:20:112
796319993912,5cyclictest141rcu_preempt20:05:132
79631999391,11cyclictest141rcu_preempt00:00:022
796315993923,12cyclictest828599-21apt-get20:40:001
796313993925,10cyclictest898831-21cron00:00:000
79631399392,33cyclictest900173-21timerwakeupswit00:00:270
79631399392,18cyclictest888476-21munin-run23:30:010
796313993921,6cyclictest13-21ksoftirqd/022:10:000
796313993921,6cyclictest13-21ksoftirqd/022:10:000
796313993916,10cyclictest13-21ksoftirqd/022:38:450
796313993914,6cyclictest141rcu_preempt23:50:280
79631399391,35cyclictest1972-21rtkit-daemon22:18:520
796313993911,13cyclictest141rcu_preempt22:30:020
796313993911,11cyclictest141rcu_preempt21:50:140
197399390,27rtkit-daemon4497-21PK-Backend19:15:122
79631999387,11cyclictest34-21ksoftirqd/221:06:422
796319993827,7cyclictest816154-21cron20:05:002
796319993825,9cyclictest34-21ksoftirqd/221:10:152
796319993815,6cyclictest141rcu_preempt19:55:142
796319993815,18cyclictest902325-21munin-run00:10:002
796319993814,6cyclictest141rcu_preempt19:45:012
796319993813,12cyclictest141rcu_preempt20:30:122
796319993813,12cyclictest141rcu_preempt20:30:112
79631599386,7cyclictest141rcu_preempt00:15:291
796315993826,7cyclictest883241-21sh23:15:011
796315993822,12cyclictest923-21dbus-daemon23:05:001
796315993819,14cyclictest26-21ksoftirqd/121:05:001
79631399389,14cyclictest882155-21df_inode23:10:140
79631399384,19cyclictest810658-21munin-run19:50:010
79631399384,19cyclictest810658-21munin-run19:50:010
796313993824,10cyclictest13-21ksoftirqd/019:15:260
79631399382,19cyclictest874318-31apt-check22:47:170
79631399382,19cyclictest868658-21users22:30:290
79631399382,19cyclictest849106-21python321:35:260
79631399382,18cyclictest876970-21diskstats22:55:160
79631399382,14cyclictest878725-21hddtemp_smartct23:00:170
796313993815,5cyclictest141rcu_preempt20:30:300
796313993815,5cyclictest141rcu_preempt20:30:290
796313993814,6cyclictest141rcu_preempt21:20:220
79631399381,32cyclictest823092-21turbostat20:25:010
79631399381,19cyclictest901585-21grep00:05:230
79631399381,19cyclictest839537-21latency_hist21:10:020
79631399381,13cyclictest886469-21python323:22:050
197399380,26rtkit-daemon4497-21PK-Backend22:49:592
79631999372,31cyclictest849705-21apt-get21:40:062
79631999372,31cyclictest849705-21apt-get21:40:062
796319993722,12cyclictest34-21ksoftirqd/221:55:022
796319993721,11cyclictest438-20systemd-journal21:35:012
796319993719,14cyclictest885997-21ntp_states23:20:252
796319993711,5cyclictest141rcu_preempt19:45:122
796319993711,5cyclictest141rcu_preempt19:45:112
796315993724,9cyclictest813605-21grep19:55:211
796315993722,11cyclictest829795-21pool-udisksd20:40:211
79631399372,20cyclictest905056-21ntp_states00:15:240
79631399372,20cyclictest796886-21df_inode19:10:140
79631399372,18cyclictest908408-21df_inode00:25:170
79631399372,18cyclictest862142-21/usr/sbin/munin22:15:010
79631399372,18cyclictest847586-21users21:30:290
79631399372,18cyclictest836702-21memory21:00:210
79631399372,18cyclictest827708-21memory20:35:210
79631399372,18cyclictest814131-40lsb_release19:56:290
79631399372,18cyclictest800839-21df_inode19:20:130
79631399372,17cyclictest821366-21munin-run20:20:010
79631399372,17cyclictest807808-21diskstats19:40:140
796313993721,11cyclictest956-21systemd-logind19:29:200
197399370,25rtkit-daemon0-21swapper/200:36:262
796319993624,7cyclictest888902-21tr23:30:122
796319993622,9cyclictest438-20systemd-journal23:43:052
796319993621,11cyclictest833899-21sshd20:52:192
796319993613,6cyclictest141rcu_preempt20:28:382
796319993613,6cyclictest141rcu_preempt20:28:382
796319993610,6cyclictest141rcu_preempt19:25:002
79631599365,6cyclictest837627-21kworker/1:021:05:291
79631599362,30cyclictest872550-21apt-get22:45:071
79631599361,9cyclictest4497-21PK-Backend21:51:481
79631399363,12cyclictest858637-21cron22:05:000
79631399362,8cyclictest834766-21diskstats20:55:150
79631399361,17cyclictest844969-21cstates21:25:130
79631999359,11cyclictest141rcu_preempt23:05:022
79631999352,10cyclictest862120-21munin-run22:15:012
796319993514,6cyclictest141rcu_preempt21:19:512
79631999351,12cyclictest141rcu_preempt22:17:402
796319993510,6cyclictest141rcu_preempt22:51:522
796319993510,6cyclictest141rcu_preempt22:51:522
796319993510,5cyclictest141rcu_preempt21:20:252
796315993528,5cyclictest0-21swapper/119:17:291
796315993519,11cyclictest912145-21cat00:35:221
79631599351,6cyclictest8810582cat23:05:281
79631599351,31cyclictest4497-21PK-Backend19:21:511
79631599351,11cyclictest141rcu_preempt21:10:191
79631599351,10cyclictest141rcu_preempt23:58:171
79631399355,14cyclictest13-21ksoftirqd/021:44:280
79631399355,14cyclictest13-21ksoftirqd/021:44:270
79631399351,12cyclictest903180-21sort00:10:210
197399350,24rtkit-daemon0-21swapper/200:03:292
796319993417,12cyclictest836341-21expr21:00:122
796319993411,5cyclictest141rcu_preempt23:25:142
79631599344,9cyclictest26-21ksoftirqd/121:17:051
79631599343,11cyclictest141rcu_preempt00:25:281
79631599341,9cyclictest4497-21PK-Backend23:52:061
79631599341,9cyclictest4497-21PK-Backend20:52:251
796315993419,11cyclictest864735-21grep22:20:231
79631599341,30cyclictest4497-21PK-Backend20:10:001
79631599341,30cyclictest4497-21PK-Backend00:07:211
79631599341,30cyclictest0-21swapper/119:35:171
79631599341,29cyclictest4497-21PK-Backend23:45:091
79631599341,28cyclictest4497-21PK-Backend23:30:471
79631399342,12cyclictest141rcu_preempt22:20:160
79631399342,12cyclictest141rcu_preempt21:55:170
796313993417,13cyclictest13-21ksoftirqd/021:11:470
79631399341,14cyclictest4497-21PK-Backend20:00:230
79631399341,11cyclictest912087-21latency00:35:220
8874502330,7chrt4497-21PK-Backend23:25:181
79631599338,4cyclictest141rcu_preempt22:15:111
79631599338,21cyclictest869793-21/usr/sbin/munin22:35:201
796315993321,8cyclictest877323-21grep22:55:211
79631599332,11cyclictest141rcu_preempt22:00:191
796315993319,9cyclictest850082-21date21:40:031
796315993318,11cyclictest884240-21ntpq23:15:241
796315993318,11cyclictest884240-21ntpq23:15:241
796315993318,11cyclictest814684-21basename20:00:121
79631599331,7cyclictest4497-21PK-Backend20:20:081
796315993317,12cyclictest903297-21awk00:10:231
796315993316,13cyclictest923-21dbus-daemon20:32:321
796315993316,13cyclictest923-21dbus-daemon20:32:321
79631599331,29cyclictest4497-21PK-Backend20:55:221
79631599331,29cyclictest4497-21PK-Backend20:11:351
79631599331,29cyclictest4497-21PK-Backend20:11:341
79631599331,29cyclictest0-21swapper/119:10:161
79631599331,28cyclictest4497-21PK-Backend21:26:561
79631599331,11cyclictest852855-21grep21:45:261
79631599331,11cyclictest141rcu_preempt23:20:551
79631399331,11cyclictest883608-21cpuspeed_turbos23:15:150
79631399331,11cyclictest883608-21cpuspeed_turbos23:15:150
79631599327,5cyclictest141rcu_preempt23:35:291
796315993214,15cyclictest4497-21PK-Backend20:20:001
79631599321,29cyclictest0-21swapper/122:07:541
79631599321,29cyclictest0-21swapper/122:07:541
79631599321,28cyclictest4497-21PK-Backend22:29:281
79631599318,5cyclictest141rcu_preempt23:42:141
796315993121,6cyclictest911272-21idleruntime-cro00:34:591
796315993119,8cyclictest809320-21tr19:45:151
796315993119,8cyclictest809320-21tr19:45:151
79631599311,11cyclictest900589-21awk00:05:011
796315993110,5cyclictest141rcu_preempt21:55:131
79631399311,14cyclictest4497-21PK-Backend21:48:100
79631599308,5cyclictest141rcu_preempt00:20:131
796315993023,4cyclictest0-21swapper/121:21:001
79631599301,14cyclictest4497-21PK-Backend22:30:231
796315992714,9cyclictest808914-21basename19:45:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional