You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-20 - 07:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Mon Apr 20, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2446529380,9sleep00-21swapper/019:06:060
2447729170,9sleep20-21swapper/219:06:162
2456929073,13sleep10-21swapper/119:07:281
2441427966,9sleep30-21swapper/319:05:293
241622310,1sleep30-21swapper/321:52:103
2485699280,26cyclictest0-21swapper/221:41:212
2485699270,21cyclictest0-21swapper/200:34:302
2484999271,11cyclictest3396-21runrttasks22:19:010
2485699250,25cyclictest0-21swapper/223:13:122
2485699250,24cyclictest0-21swapper/221:17:222
24859992422,2cyclictest39-21ksoftirqd/322:00:463
24856992419,4cyclictest17435-21diskmemload22:05:392
2485699240,24cyclictest0-21swapper/200:04:142
2484999244,6cyclictest31504-21strings22:35:200
2484999244,2cyclictest0-21swapper/022:10:140
2485999230,3cyclictest0-21swapper/322:20:153
2485699234,17cyclictest0-21swapper/200:25:342
2485399220,22cyclictest0-21swapper/122:11:461
2484999223,3cyclictest0-21swapper/022:20:280
2485699210,21cyclictest0-21swapper/223:00:362
2485699210,18cyclictest0-21swapper/200:08:392
24853992118,2cyclictest19192-21tr20:05:151
2484999212,3cyclictest29608-21sendmail19:20:000
2485399200,15cyclictest17435-21diskmemload23:00:261
2484999201,2cyclictest21378-21aten2_r7power_p20:10:150
2485999190,2cyclictest0-21swapper/300:39:013
24856991910,9cyclictest0-21swapper/223:44:372
2485699190,19cyclictest0-21swapper/223:27:592
2485699190,18cyclictest0-21swapper/222:55:472
2485399190,1cyclictest0-21swapper/100:27:071
2484999190,4cyclictest0-21swapper/022:54:160
2484999190,2cyclictest0-21swapper/022:31:210
2484999190,2cyclictest0-21swapper/022:31:210
2485699180,18cyclictest0-21swapper/223:56:042
2485699180,16cyclictest0-21swapper/222:11:522
2485399184,12cyclictest17435-21diskmemload00:14:041
24853991816,2cyclictest25-21ksoftirqd/122:03:481
24853991815,3cyclictest0-21swapper/121:18:541
2485399181,1cyclictest17435-21diskmemload00:20:371
2485399180,2cyclictest0-21swapper/123:46:271
2485399180,17cyclictest0-21swapper/100:15:571
2484999181,8cyclictest14570-21ssh23:42:480
24849991815,2cyclictest12332-21ssh00:20:140
24849991810,7cyclictest6930-21tail22:40:180
24849991810,7cyclictest27082-21cat20:20:210
2484999180,14cyclictest0-21swapper/023:19:080
164992180,0sleep30-21swapper/323:44:193
2485699171,11cyclictest15216-21sensors22:25:252
2485699170,17cyclictest0-21swapper/222:21:482
2485399170,1cyclictest17435-21diskmemload23:08:501
2485399170,17cyclictest26191-21ssh21:53:461
2485399170,17cyclictest0-21swapper/123:34:301
2485399170,17cyclictest0-21swapper/122:18:461
2484999179,7cyclictest10397-21grep23:20:270
2484999170,5cyclictest0-21swapper/021:11:240
2484999170,3cyclictest0-21swapper/000:07:240
2484999170,16cyclictest9532-21ssh23:59:350
2485699164,10cyclictest1775-21python23:15:242
24856991611,3cyclictest7913-21wc19:40:182
2485699160,15cyclictest0-21swapper/200:37:212
24853991612,3cyclictest0-21swapper/100:35:571
2485399160,1cyclictest0-21swapper/121:36:381
2485399160,16cyclictest0-21swapper/123:56:411
2484999169,6cyclictest15633-21cstates21:05:150
2484999168,2cyclictest24587-21ssh00:26:570
2484999168,1cyclictest8784-21cat20:50:190
2484999167,2cyclictest17672-21ssh21:48:380
2484999167,2cyclictest16514-21grep23:05:180
2484999160,6cyclictest0-21swapper/021:42:360
2484999160,12cyclictest1196-21runrttasks21:58:040
24859991513,1cyclictest0-21swapper/300:00:003
2485699153,3cyclictest225-21systemd-journal19:21:532
2485699150,2cyclictest0-21swapper/222:38:332
2485399152,9cyclictest26906-21df_inode23:50:151
24853991514,1cyclictest38150irq/131-snd_hda21:40:321
2485399150,1cyclictest0-21swapper/119:25:231
2485399150,15cyclictest0-21swapper/123:27:431
2485399150,14cyclictest0-21swapper/123:35:181
2485399150,14cyclictest0-21swapper/123:19:311
2484999158,6cyclictest2863-21sh20:40:010
2484999158,2cyclictest30308-21cut00:30:210
2484999158,2cyclictest25343-21rm21:53:060
2484999158,2cyclictest19257-21ssh22:28:180
2484999158,2cyclictest13239-21taskset21:25:560
2484999158,1cyclictest390-21ssh00:12:490
2484999158,1cyclictest32567-21expr19:25:160
2484999158,1cyclictest20605-21rm23:26:450
2484999158,1cyclictest19170-21cut23:45:200
2484999157,2cyclictest5253-21sh20:45:000
2484999157,2cyclictest31577-21cat22:55:140
2484999157,2cyclictest31560-21head19:20:270
2484999151,5cyclictest26270-21apt-get21:15:130
2484999151,4cyclictest0-21swapper/023:50:130
24849991511,4cyclictest94-21kswapd023:00:130
2484999150,9cyclictest70992sleep022:01:080
2484999150,9cyclictest25829-21date20:20:000
2484999150,10cyclictest9450-21rm00:18:210
2484999150,10cyclictest12392-21sendmail_mailqu00:00:240
24859991410,3cyclictest8851-21tr23:20:163
24859991410,2cyclictest28232-21wc20:25:013
24859991410,2cyclictest10694-21grep21:25:153
2485999140,1cyclictest141rcu_preempt21:58:273
2485699148,5cyclictest11667-21apache221:24:562
24856991410,3cyclictest16866-21wc23:05:202
2485699140,4cyclictest0-21swapper/222:30:522
2485699140,4cyclictest0-21swapper/222:30:512
2485699140,14cyclictest0-21swapper/223:39:322
2485699140,0cyclictest0-21swapper/223:46:102
2485399144,9cyclictest15248-21python22:25:251
2485399142,6cyclictest0-21swapper/120:33:111
24853991410,3cyclictest3409-21ssh22:38:141
24853991410,3cyclictest0-21swapper/100:07:361
2485399140,14cyclictest0-21swapper/122:46:321
2485399140,13cyclictest0-21swapper/122:52:421
2484999148,2cyclictest6101-21sh21:21:550
2484999148,1cyclictest21999-21taskset21:31:160
2484999147,2cyclictest17126-21ssh22:45:560
2484999147,2cyclictest1034-21sh21:38:440
2484999147,1cyclictest6922-21sed00:35:280
2484999147,1cyclictest30833-21cut20:30:150
2484999146,2cyclictest26546-21ssh23:30:200
2484999140,9cyclictest17256-21sh22:07:320
2485999139,3cyclictest32305-21cut19:25:153
2485999139,3cyclictest2658-21awk21:40:003
2485999135,4cyclictest225-21systemd-journal21:49:533
2485999131,6cyclictest0-21swapper/300:04:013
24859991311,1cyclictest11667-21apache221:19:113
24859991311,1cyclictest0-21swapper/321:22:273
24859991310,2cyclictest2870-21awk20:40:003
24859991310,2cyclictest24108-21tr22:50:213
24859991310,2cyclictest19815-21sed21:30:193
24859991310,2cyclictest17765-21grep23:25:173
2485999130,13cyclictest0-21swapper/322:25:463
2485699139,3cyclictest27392-21cat23:50:192
24856991311,0cyclictest0-21swapper/223:34:242
24856991310,2cyclictest28169-21grep19:15:162
24856991310,2cyclictest256992sleep222:51:072
24856991310,2cyclictest17748-21grep20:00:222
24856991310,2cyclictest15048-21sed22:45:152
2485699130,13cyclictest0-21swapper/223:20:422
2485699130,13cyclictest0-21swapper/222:19:222
2485699130,13cyclictest0-21swapper/221:53:032
2485699130,11cyclictest0-21swapper/200:10:132
2485399139,3cyclictest2945-21head19:30:171
2485399139,3cyclictest28686-21tr20:25:161
2485399139,2cyclictest28216-21grep19:15:161
2485399133,3cyclictest0-21swapper/121:58:321
24853991310,2cyclictest19016-21grep21:30:141
2485399130,13cyclictest0-21swapper/122:41:001
2484999137,2cyclictest28388-21taskset20:25:060
2484999137,1cyclictest6107-21entropy20:45:160
2484999137,1cyclictest39842sleep019:31:060
2484999137,1cyclictest28039-21ssh23:12:010
2484999137,1cyclictest17615-21uname20:00:200
2484999137,1cyclictest14708-21taskset21:03:340
2484999137,1cyclictest14135-21apache_processe19:55:140
2484999133,2cyclictest2999-21ssh23:35:260
2485999129,2cyclictest5664-21head19:35:193
2485999129,2cyclictest4845-21head20:40:263
2485999129,2cyclictest3317-21tr19:30:203
2485999129,2cyclictest31692-21wc22:55:153
2485999129,2cyclictest31522-21head20:30:183
2485999129,2cyclictest25345-21grep19:10:133
2485999129,2cyclictest18259-21tr21:10:153
2485999129,2cyclictest17370-21munin-node22:43:403
2485999129,2cyclictest1110-21chrt23:54:323
2485999128,3cyclictest30221-21apache_accesses22:35:123
2485999128,3cyclictest28142-21tr00:10:153
2485999128,3cyclictest17642-21grep23:05:273
2485999125,3cyclictest55650irq/127-enp2s0-00:27:553
2485999120,11cyclictest7966-21cron23:20:003
2485699129,2cyclictest5997-21grep22:00:262
2485699129,2cyclictest28376-21tr21:55:142
2485699129,2cyclictest24434-21sed20:15:182
2485699129,2cyclictest19785-21sed21:30:192
2485699129,2cyclictest13825-21grep21:00:222
2485699129,2cyclictest10262-21uname19:45:182
2485699128,3cyclictest6233-21tail20:45:182
2485699128,3cyclictest12274-21grep21:45:162
2485699128,2cyclictest18695-21cut21:10:172
2485699122,4cyclictest0-21swapper/200:25:002
2485699120,12cyclictest0-21swapper/221:35:172
2485699120,11cyclictest21922-21grep20:10:172
2485399129,2cyclictest8544-21chrt21:23:521
2485399129,2cyclictest5177-21cut19:35:151
2485399129,2cyclictest18461-21expr21:10:151
2485399129,2cyclictest13945-21grep21:00:241
2485399129,2cyclictest13863-21cron19:55:001
2485399129,2cyclictest12041-21head20:55:271
2485399129,2cyclictest10415-21grep19:45:181
2485399128,3cyclictest7982-21wc19:40:191
2485399128,3cyclictest24722-21cut23:10:161
2485399128,3cyclictest2266-21wc20:35:271
2485399128,3cyclictest17763-21grep20:00:221
2485399127,4cyclictest7861-21taskset20:50:081
2485399121,10cyclictest0-21swapper/119:55:151
2485399120,1cyclictest17435-21diskmemload22:59:251
2484999129,2cyclictest7059-21tr19:40:140
2484999129,2cyclictest25327-21grep19:10:130
2484999129,2cyclictest11366-21cat20:55:200
2484999127,1cyclictest6528-21taskset19:38:210
2484999123,8cyclictest11429-21cron19:50:000
2484999120,4cyclictest225-21systemd-journal20:10:000
184632121,1sleep30-21swapper/322:47:013
2485999118,2cyclictest7932-21grep20:50:143
2485999118,2cyclictest7117-21ssh23:00:083
2485999118,2cyclictest6794-21ssh21:41:413
2485999118,2cyclictest6654-21grep20:45:213
2485999118,2cyclictest6202-21ssh00:15:493
2485999118,2cyclictest4137-21ssh23:36:123
2485999118,2cyclictest31960-21cron19:24:593
2485999118,2cyclictest29430-21chrt19:18:193
2485999118,2cyclictest27045-21ssh23:11:153
2485999118,2cyclictest26643-21ssh22:13:213
2485999118,2cyclictest26288-21cut23:30:183
2485999118,2cyclictest23345-21chrt20:14:413
2485999118,2cyclictest22058-21sh22:30:163
2485999118,2cyclictest22058-21sh22:30:153
2485999118,2cyclictest20412-21grep00:05:213
2485999118,2cyclictest14418-21sed21:00:293
2485999118,2cyclictest13944-21grep22:05:203
2485999118,2cyclictest12476-21tr21:00:013
2485999118,2cyclictest12362-21uniq19:50:163
2485999111,9cyclictest32170-21ssh22:16:093
24859991111,0cyclictest0-21swapper/300:24:153
2485999110,11cyclictest0-21swapper/323:46:453
2485699118,2cyclictest8067-21sed22:40:282
2485699118,2cyclictest782-21tr20:35:152
2485699118,2cyclictest3444-21wc19:30:222
2485699118,2cyclictest32505-21tr19:25:152
2485699118,2cyclictest25730-21cut19:10:172
2485699118,2cyclictest19157-21tr20:05:152
2485699118,2cyclictest15559-21grep21:05:152
2485699118,2cyclictest15225-21cut19:55:212
2485699118,2cyclictest11791-21hostname19:50:142
2485699118,2cyclictest10961-21expr21:25:162
2485699117,3cyclictest4849-21grep20:40:262
2485699111,8cyclictest489-21cron19:40:002
2485699110,0cyclictest0-21swapper/200:15:572
2485399118,2cyclictest9756-21tr23:40:141
2485399118,2cyclictest6702-21ssh22:20:251
2485399118,2cyclictest529-21ssh00:31:561
2485399118,2cyclictest22531-21wc20:10:231
2485399118,2cyclictest21770-21ssh22:30:061
2485399118,2cyclictest21770-21ssh22:30:061
2485399118,2cyclictest16994-21grep21:05:261
2485399118,2cyclictest16742-21ssh21:47:551
2485399118,2cyclictest11705-21head00:00:191
2485399117,3cyclictest25412-21taskset20:15:381
2485399116,4cyclictest0-21swapper/123:23:451
2484999117,3cyclictest13417-21sed19:50:260
2485999108,1cyclictest9666-21expr19:45:133
2485999108,1cyclictest28733-21cat20:25:163
2485999108,1cyclictest25674-21chrt20:18:583
2485999108,1cyclictest15649-21cat19:55:283
2485999107,2cyclictest6982-21cut19:40:133
2485999107,2cyclictest29052-21bc00:30:123
2485999107,2cyclictest19445-21df_abs20:05:153
2485999107,2cyclictest16094-21grep21:05:173
2485999100,9cyclictest16813-21cut20:00:153
2485699108,1cyclictest30816-21hostname20:30:142
2485699108,1cyclictest11384-21grep20:55:202
2485699107,2cyclictest8144-21tr20:50:152
2485699107,2cyclictest30192-21chrt20:25:462
2485699107,2cyclictest27004-21cut20:20:212
2485399108,1cyclictest27879-21taskset20:21:311
2485399108,1cyclictest11208-21ssh21:25:171
2485399107,2cyclictest25126-21basename19:10:131
2485399107,2cyclictest14953-21ssh22:05:431
2485399100,6cyclictest225-21systemd-journal20:46:371
2485399100,5cyclictest31971-21munin-run19:24:591
248539996,2cyclictest4168-21load20:40:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional