You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-22 - 23:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sun Mar 22, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2820029683,9sleep00-21swapper/007:05:570
2824228472,8sleep20-21swapper/207:06:322
2839828269,9sleep30-21swapper/307:08:333
2824127658,14sleep10-21swapper/107:06:311
17382630,0sleep20-21swapper/210:34:402
28598993223,8cyclictest21408-21diskmemload12:25:081
2859899280,1cyclictest0-21swapper/109:50:221
2859899260,3cyclictest0-21swapper/111:05:181
2859899251,23cyclictest0-21swapper/111:03:501
2859899240,23cyclictest0-21swapper/109:58:441
2860399230,20cyclictest0-21swapper/211:07:162
2859899230,4cyclictest17620-21cron07:55:001
2859899230,4cyclictest0-21swapper/109:20:111
2859899230,3cyclictest0-21swapper/112:04:181
2859899230,3cyclictest0-21swapper/110:30:471
2859899230,3cyclictest0-21swapper/110:25:201
2859899230,2cyclictest21408-21diskmemload11:30:251
2859899230,22cyclictest1-21systemd12:20:001
2860399220,22cyclictest0-21swapper/211:12:582
2860399220,22cyclictest0-21swapper/210:01:312
2859899228,6cyclictest25247-21ssh12:05:321
2859899224,4cyclictest25899-21kworker/1:211:50:151
2859899223,1cyclictest21408-21diskmemload12:31:081
2859899222,3cyclictest21408-21diskmemload11:44:471
2859899222,1cyclictest21408-21diskmemload10:11:201
2859899220,1cyclictest0-21swapper/111:55:531
2859499220,2cyclictest0-21swapper/012:15:390
289422210,0sleep00-21swapper/011:29:270
2860599210,21cyclictest0-21swapper/311:51:103
2860399210,21cyclictest0-21swapper/211:53:552
2860399210,21cyclictest0-21swapper/209:27:352
2859899213,2cyclictest25261-21grep09:30:251
2859899213,1cyclictest21408-21diskmemload09:44:301
2859899210,2cyclictest0-21swapper/110:57:391
2859899210,2cyclictest0-21swapper/108:45:151
2859499210,1cyclictest0-21swapper/009:57:280
28605992020,0cyclictest0-21swapper/312:20:303
28603992018,1cyclictest0-21swapper/210:22:242
2860399200,20cyclictest0-21swapper/210:55:202
2859899201,3cyclictest24026-21sh09:50:001
2859899201,18cyclictest0-21swapper/107:45:151
2859899200,5cyclictest0-21swapper/109:35:591
2859899200,17cyclictest3406-21ssh11:14:221
2860599191,11cyclictest0-21swapper/307:35:123
2860599190,5cyclictest0-21swapper/309:59:043
2860599190,19cyclictest0-21swapper/311:14:003
2860399190,19cyclictest0-21swapper/212:09:162
2859899196,1cyclictest0-21swapper/110:16:401
2859899193,4cyclictest29030-21sh11:29:321
2859899192,2cyclictest21408-21diskmemload10:20:451
2859899190,4cyclictest0-21swapper/109:25:101
2859899190,17cyclictest0-21swapper/110:50:141
2859499190,3cyclictest0-21swapper/011:21:380
2860599180,15cyclictest0-21swapper/309:14:323
2860399181,17cyclictest0-21swapper/209:12:132
2860399180,18cyclictest0-21swapper/212:25:412
2859899186,7cyclictest7995-21ssh11:35:281
2859899181,2cyclictest0-21swapper/110:37:061
28598991810,7cyclictest20978-21rm10:45:251
2859899180,18cyclictest0-21swapper/110:09:311
2859499183,9cyclictest0-21swapper/011:15:130
28605991716,1cyclictest0-21swapper/309:51:103
2860399170,15cyclictest0-21swapper/209:31:392
2859899179,7cyclictest9049-21sed08:40:251
2859899179,7cyclictest28030-21grep08:15:171
2859899177,9cyclictest0-21swapper/112:10:141
2859899171,3cyclictest0-21swapper/109:00:201
2859899171,3cyclictest0-21swapper/109:00:201
2859499174,6cyclictest11511-21munin-run11:00:010
2860599164,5cyclictest23059-21grep11:25:213
2860599160,16cyclictest0-21swapper/312:28:473
2860599160,16cyclictest0-21swapper/312:09:563
2860599160,16cyclictest0-21swapper/311:36:063
2860599160,16cyclictest0-21swapper/310:20:423
28603991614,1cyclictest0-21swapper/209:40:062
2860399160,2cyclictest0-21swapper/211:33:262
2860399160,13cyclictest0-21swapper/212:13:232
2859899168,7cyclictest9134-21basename07:35:161
2859899166,7cyclictest13104-21apt-get11:20:021
2859899160,14cyclictest0-21swapper/111:48:151
28594991614,2cyclictest0-21swapper/010:46:410
2860599151,14cyclictest0-21swapper/310:09:073
2860599150,0cyclictest0-21swapper/310:34:543
2860399150,15cyclictest0-21swapper/210:05:522
2859899158,6cyclictest344-21sort08:25:161
2859899158,6cyclictest15813-21ssh12:39:571
2859899157,7cyclictest8414-21ssh11:16:451
2859899157,7cyclictest31348-21wc08:20:251
2859899157,7cyclictest15523-21rm10:42:171
2859899157,7cyclictest15523-21rm10:42:171
2859899157,7cyclictest12952-21rm10:02:071
2859899156,8cyclictest23369-21gzip09:10:211
2859899153,1cyclictest205422grep08:00:151
2859899152,2cyclictest225-21systemd-journal08:00:001
2859499153,10cyclictest27734-21/usr/sbin/munin10:50:210
2859499150,1cyclictest0-21swapper/012:33:130
2859499150,0cyclictest0-21swapper/010:04:220
2860599146,2cyclictest32034-21sendmail-mta11:30:263
2860599143,7cyclictest23738-21taskset10:47:153
2860599142,2cyclictest31707-21ssh09:35:113
28605991411,2cyclictest8180-21grep12:15:173
28605991411,2cyclictest25290-21tr08:10:143
2860599140,1cyclictest0-21swapper/310:02:453
2860599140,14cyclictest0-21swapper/310:43:043
2860599140,14cyclictest0-21swapper/310:43:033
2860599140,13cyclictest0-21swapper/312:02:243
28603991413,1cyclictest32731-21/usr/sbin/munin09:55:182
28603991411,2cyclictest14015-21apache209:35:582
28603991411,2cyclictest10698-21wc08:45:212
2860399141,12cyclictest0-21swapper/211:00:532
28603991410,3cyclictest947-21turbostat10:15:002
2860399140,2cyclictest0-21swapper/209:19:272
2860399140,14cyclictest0-21swapper/211:26:272
2860399140,13cyclictest0-21swapper/207:20:122
2859899148,5cyclictest16601-21tune2fs12:20:161
2859899148,5cyclictest1379-21ssh09:16:251
2859899147,6cyclictest6739-21chrt08:38:581
2859899147,6cyclictest22947-21grep08:05:151
2859899146,7cyclictest13233-21cat08:50:221
2859899142,5cyclictest2685-21grep08:30:151
2859899141,2cyclictest225-21systemd-journal07:20:001
2859899140,13cyclictest5856-21cat07:30:011
2859499149,2cyclictest27491-21grep10:10:270
28594991413,1cyclictest0-21swapper/011:45:190
28594991411,2cyclictest16788-21ssh10:24:130
2859499140,2cyclictest0-21swapper/012:04:570
294932130,1sleep00-21swapper/009:33:400
2860599139,3cyclictest5006-21tr10:55:213
2860599139,3cyclictest12861-21head08:50:183
2860599133,5cyclictest225-21systemd-journal07:17:003
2860599132,9cyclictest7853-21ntp_kernel_pll_11:55:203
28605991312,1cyclictest0-21swapper/309:48:573
28605991310,2cyclictest32595-21tr12:30:143
28605991310,2cyclictest26295-21cron10:30:003
28605991310,2cyclictest23392-21head08:05:173
28605991310,2cyclictest15393-21wc07:50:003
28605991310,2cyclictest11317-21grep08:45:273
2860399139,3cyclictest14626-21tr11:20:192
2860399138,3cyclictest19209-21wc07:55:232
28603991312,1cyclictest28585-21cyclictest10:53:162
28603991312,1cyclictest0-21swapper/212:21:312
28603991310,2cyclictest8493-21grep11:55:262
28603991310,2cyclictest7174-21tail11:35:182
28603991310,2cyclictest2752-21grep10:35:142
2860399130,2cyclictest0-21swapper/210:43:592
2860399130,2cyclictest0-21swapper/210:43:592
2860399130,13cyclictest0-21swapper/211:40:332
2860399130,13cyclictest0-21swapper/210:29:142
2860399130,13cyclictest0-21swapper/209:46:322
2859899136,6cyclictest12345-21sed07:40:261
2859899132,10cyclictest2102-21/usr/sbin/munin07:20:171
2859899131,4cyclictest15162-21fschecks_time08:55:161
2859899130,4cyclictest29526-21fschecks_time07:10:171
2859499131,6cyclictest14270-21awk08:54:590
28594991312,1cyclictest0-21swapper/011:12:580
28594991310,2cyclictest24172-21cat09:50:000
28594991310,2cyclictest23434-21grep12:05:140
28594991310,2cyclictest2289-21ssh10:35:000
28594991310,2cyclictest18967-21wc10:05:260
2859499130,7cyclictest31943-21perf08:25:000
2859499130,13cyclictest0-21swapper/011:01:540
2860599129,2cyclictest9630-21ssh12:35:223
2860599129,2cyclictest8109-21cut08:40:163
2860599129,2cyclictest4319-21sed07:25:193
2860599129,2cyclictest31911-21cron08:25:003
2860599129,2cyclictest23647-21grep09:30:143
2860599128,3cyclictest8306-21cat09:20:243
2860599128,3cyclictest13881-21grep11:00:253
2860599128,2cyclictest21258-21head11:05:173
2860599128,2cyclictest18970-21ssh11:23:123
2860599128,2cyclictest1500-21grep10:15:123
2860599125,3cyclictest225-21systemd-journal07:14:593
2860399129,2cyclictest8955-21grep12:35:182
2860399129,2cyclictest32307-21grep07:15:212
2860399129,2cyclictest32300-21munin-plugin-st09:55:002
2860399129,2cyclictest29846-21grep08:20:132
2860399129,2cyclictest29600-21sed07:10:182
2860399129,2cyclictest27898-21grep08:15:162
2860399129,2cyclictest26227-21sed08:10:202
2860399129,2cyclictest10742-21cut07:40:142
2860399122,9cyclictest16538-21sensors12:00:242
2860399122,0cyclictest141rcu_preempt09:20:092
28603991211,1cyclictest32-21ksoftirqd/211:16:102
2859899127,4cyclictest7678-21awk07:30:261
2859899123,2cyclictest20520-21grep09:05:241
2859899121,2cyclictest225-21systemd-journal08:12:441
2859499129,2cyclictest9095-21tail07:35:150
2859499129,2cyclictest8431-21grep09:20:240
2859499129,2cyclictest7384-21wc11:35:210
2859499129,2cyclictest21098-21ls08:00:170
2859499129,2cyclictest2044-21df10:15:150
2859499129,2cyclictest17727-21grep12:20:250
2859499129,2cyclictest16346-21head09:25:200
2859499129,2cyclictest15744-21grep08:55:210
2859499129,2cyclictest15381-21grep11:40:190
2859499129,2cyclictest10002-21grep12:35:270
2859499128,2cyclictest0-21swapper/011:07:160
2859499124,4cyclictest30036-21apt-get11:30:130
28594991210,1cyclictest707-21ssh09:35:210
28594991210,1cyclictest20393-21grep10:25:250
2859499120,11cyclictest32499-21sed09:15:250
2860599118,2cyclictest7803-21rm09:40:133
2860599118,2cyclictest751-21find12:10:233
2860599118,2cyclictest3485-21grep10:35:173
2860599118,2cyclictest30438-21ssh09:15:103
2860599118,2cyclictest29994-21ssh10:12:233
2860599118,2cyclictest2762-21grep08:30:163
2860599118,2cyclictest23282-21ssh11:45:173
2860599118,2cyclictest20352-21grep08:00:143
2860599118,2cyclictest15674-21cut09:25:153
2860599118,2cyclictest15603-21tail08:55:203
2860599118,2cyclictest13945-21ssh11:40:063
2860599118,2cyclictest11330-21grep07:40:173
2860599117,3cyclictest17856-21/usr/sbin/munin07:55:183
2860599112,6cyclictest7560-21grep07:30:253
2860399118,2cyclictest9338-21wc07:35:182
2860399118,2cyclictest9027-21head08:40:252
2860399118,2cyclictest878-21wc12:30:192
2860399118,2cyclictest3667-21grep07:25:162
2860399118,2cyclictest24677-21taskset11:45:392
2860399118,2cyclictest24510-21taskset08:06:542
2860399118,2cyclictest20966-21sed10:45:252
2860399118,2cyclictest16629-21ls09:00:002
2860399118,2cyclictest13688-21sed08:50:272
2860399118,2cyclictest13361-21tr07:45:162
2860399118,2cyclictest10372-21ssh12:16:102
2860399117,3cyclictest3367-21tail10:15:262
2859499119,1cyclictest32112-21ls11:50:210
2859499119,1cyclictest24865-21grep12:25:170
2859499119,1cyclictest13861-21head07:45:190
2859499118,2cyclictest8488-21ntpq08:40:200
2859499118,2cyclictest6885-21tr11:55:130
2859499118,2cyclictest6342-21rm12:14:330
2859499118,2cyclictest32310-21grep07:15:210
2859499118,2cyclictest2905-21sed07:20:250
2859499118,2cyclictest24803-21grep09:50:140
2859499118,2cyclictest2403-21tr08:30:140
2859499118,2cyclictest22145-21grep09:10:140
2859499118,2cyclictest11161-21hostname10:40:140
2859499118,2cyclictest11161-21hostname10:40:140
2860599108,1cyclictest321412sleep308:25:023
2860599108,1cyclictest285212sleep308:15:203
2860599108,1cyclictest27695-21rm10:50:103
2860599108,1cyclictest18976-21taskset09:04:343
2860599108,1cyclictest18976-21taskset09:04:343
2860599108,1cyclictest16487-21cat07:50:203
2860599107,2cyclictest4554-21mktemp08:35:003
2860599107,2cyclictest19921-21grep09:05:183
2860599107,2cyclictest1733-21df07:20:133
2860599101,5cyclictest0-21swapper/311:15:093
2860399108,1cyclictest5886-21ntpq08:35:212
2860399108,1cyclictest1448-21head08:25:262
2860399107,2cyclictest3364-21tr08:30:212
2860399107,2cyclictest20979-21perl08:00:172
2860399107,2cyclictest20028-21grep09:05:192
2860399107,2cyclictest18678-21seq09:00:452
2860399107,2cyclictest18678-21seq09:00:442
2860399101,5cyclictest0-21swapper/207:50:142
2859499108,1cyclictest5896-21seq07:30:050
2859499108,1cyclictest4537-21apt-key08:35:000
2859499108,1cyclictest3198-21expr10:35:150
2859499108,1cyclictest25676-21fschecks_time08:10:160
2859499108,1cyclictest17505-21fschecks_count09:00:160
2859499108,1cyclictest17505-21fschecks_count09:00:160
2859499108,1cyclictest1560-21chrt08:26:060
2859499107,2cyclictest5660-21cron07:30:000
2859499107,2cyclictest19299-21taskset09:05:160
2859499100,9cyclictest18812-21kernelversion07:55:200
2859499100,7cyclictest23391-21tail08:05:170
2859499100,6cyclictest8130-21cat09:40:140
286039997,1cyclictest6155-21tr07:30:142
285949998,1cyclictest17455-21chrt07:53:010
285949997,2cyclictest11589-21chrt08:47:150
285949997,1cyclictest29520-21cron08:20:000
285949996,2cyclictest29102-21grep07:10:150
285949996,2cyclictest10846-21tr07:40:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional