You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-17 - 23:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon Aug 17, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1437028965,6sleep00-21swapper/007:08:180
1435328760,9sleep30-21swapper/307:08:043
1428328270,8sleep20-21swapper/207:07:112
1437627758,15sleep10-21swapper/107:08:221
252052720,0sleep10-21swapper/107:30:171
14590993130,1cyclictest0-21swapper/211:57:132
1458799280,2cyclictest0-21swapper/109:34:181
1459099260,24cyclictest0-21swapper/209:15:102
1459099260,24cyclictest0-21swapper/209:15:102
1459599241,3cyclictest0-21swapper/312:31:413
1459099230,22cyclictest0-21swapper/210:43:122
1458299230,2cyclictest0-21swapper/010:49:350
1459599220,0cyclictest0-21swapper/310:18:253
14587992221,1cyclictest0-21swapper/110:38:251
1458799203,16cyclictest30329-21ssh11:38:131
1458799200,19cyclictest0-21swapper/111:42:031
1459599190,19cyclictest0-21swapper/311:31:023
1459099192,1cyclictest30027-21perl07:40:172
1459099190,17cyclictest0-21swapper/209:10:592
14582991918,1cyclictest0-21swapper/011:12:070
14595991816,2cyclictest39-21ksoftirqd/309:13:163
1459099182,0cyclictest141rcu_preempt12:38:412
1459099180,17cyclictest0-21swapper/211:32:102
1458299183,13cyclictest8502-21munin-node12:00:140
1458299180,4cyclictest0-21swapper/011:30:140
1458299180,2cyclictest0-21swapper/009:19:200
1458299180,2cyclictest0-21swapper/009:19:190
1458299180,17cyclictest0-21swapper/012:15:130
1459599173,11cyclictest17207-21apt-get07:15:133
14595991710,5cyclictest39-21ksoftirqd/311:29:073
1459099172,7cyclictest3318-21apt-get09:00:132
14590991714,2cyclictest32-21ksoftirqd/208:55:002
14590991712,5cyclictest1528-21apache210:48:012
1459099170,1cyclictest141rcu_preempt09:45:152
1459099170,14cyclictest0-21swapper/209:41:232
14587991716,1cyclictest0-21swapper/112:18:111
14587991716,1cyclictest0-21swapper/109:26:441
1458799170,16cyclictest6936-21/usr/sbin/munin11:05:181
1458299170,2cyclictest0-21swapper/009:32:330
14595991616,0cyclictest0-21swapper/312:04:043
1459599160,2cyclictest0-21swapper/311:03:473
1459599160,15cyclictest0-21swapper/311:47:363
1459099164,8cyclictest537-21snmpd10:35:512
14590991613,2cyclictest26763-21latency_hist09:59:592
1459099160,0cyclictest0-21swapper/209:29:432
1458799163,1cyclictest7806-21diskmemload09:36:521
14587991612,1cyclictest25-21ksoftirqd/112:14:451
1458799161,15cyclictest7806-21diskmemload11:47:501
1458799161,11cyclictest141rcu_preempt10:49:201
1458799160,16cyclictest0-21swapper/112:25:011
14582991616,0cyclictest0-21swapper/010:26:300
1458299160,4cyclictest0-21swapper/010:37:110
1459599150,1cyclictest0-21swapper/309:47:263
1459599150,14cyclictest0-21swapper/311:43:203
1459599150,14cyclictest0-21swapper/310:27:333
14590991514,1cyclictest55650irq/127-enp2s0-12:18:112
14590991514,1cyclictest0-21swapper/208:25:142
1459099151,13cyclictest12404-21df10:10:182
1459099150,15cyclictest0-21swapper/210:57:432
1459099150,14cyclictest0-21swapper/212:20:112
14587991514,1cyclictest0-21swapper/109:12:431
14587991511,1cyclictest25-21ksoftirqd/109:46:381
1458799151,0cyclictest141rcu_preempt09:40:521
1458799150,3cyclictest0-21swapper/111:34:181
1458799150,15cyclictest0-21swapper/111:19:401
1458799150,13cyclictest0-21swapper/109:50:111
1458299155,5cyclictest28014-21perf10:20:000
1458299154,6cyclictest11184-21apt-get08:09:590
1459599140,13cyclictest0-21swapper/312:26:473
14590991413,1cyclictest32-21ksoftirqd/210:22:382
14590991410,3cyclictest13278-21grep12:25:282
14590991410,2cyclictest32447-21grep07:45:172
1458799146,2cyclictest225-21systemd-journal09:08:031
14587991412,2cyclictest25-21ksoftirqd/110:52:521
14587991410,3cyclictest26859-21wc10:00:001
14587991410,1cyclictest25-21ksoftirqd/111:57:451
1458799140,13cyclictest26737-21grep08:40:181
1458299140,1cyclictest0-21swapper/012:30:240
1459599139,4cyclictest1528-21apache212:38:313
1459599131,0cyclictest0-21swapper/309:18:243
1459599131,0cyclictest0-21swapper/309:18:233
14595991310,2cyclictest2594-21ssh11:21:083
14595991310,2cyclictest17999-21perf12:10:003
14595991310,2cyclictest1703-21grep08:55:163
1459599130,12cyclictest23418-21ls07:25:263
1459099139,3cyclictest9302-21tr08:05:152
1459099139,3cyclictest29950-21sed08:45:292
1459099139,3cyclictest1023-21cut11:40:142
14590991310,2cyclictest9464-21grep09:30:142
14590991310,2cyclictest7958-21wc08:00:232
14590991310,2cyclictest3334-21sed07:50:272
14590991310,2cyclictest16305-21wc07:10:272
1459099130,12cyclictest5760-21tr10:25:202
1459099130,12cyclictest26361-21grep09:20:202
1459099130,12cyclictest25459-21wc11:15:252
1459099130,12cyclictest0-21swapper/211:37:472
1458799139,3cyclictest24547-21wc08:35:191
1458799139,3cyclictest11022-21cut12:05:181
1458799136,3cyclictest225-21systemd-journal12:05:001
1458799132,5cyclictest3529-21sh12:40:001
1458799132,2cyclictest225-21systemd-journal08:55:001
14587991312,1cyclictest0-21swapper/108:30:151
14587991310,2cyclictest2900-21wc10:05:011
14587991310,2cyclictest22640-21basename10:55:131
14587991310,2cyclictest19306-21cron07:20:011
1458799130,1cyclictest7806-21diskmemload10:27:011
1458299139,3cyclictest25361-21wc11:15:250
1458299138,4cyclictest17543-21tr11:50:150
1458299137,5cyclictest3368-21ssh10:05:130
14582991310,2cyclictest7816-21grep11:05:170
14582991310,2cyclictest31450-21munin-run07:45:000
14582991310,2cyclictest28538-21grep07:35:250
14582991310,2cyclictest11990-21grep12:25:170
1458299130,2cyclictest0-21swapper/009:47:340
1458299130,13cyclictest0-21swapper/010:12:310
235312120,1sleep30-21swapper/311:15:073
1459599129,2cyclictest7333-21grep10:45:223
1459599129,2cyclictest6926-21grep08:00:153
1459599129,2cyclictest32737-21sort07:45:203
1459599129,2cyclictest31918-21grep08:50:223
1459599129,2cyclictest26115-21ssh09:20:183
1459599129,2cyclictest24950-21uniq07:30:163
1459599129,2cyclictest24043-21df08:35:163
1459599129,2cyclictest22210-21ssh09:56:273
1459599129,2cyclictest20430-21sed12:10:263
1459599129,2cyclictest18518-21cat09:55:013
1459599129,2cyclictest16259-21grep10:50:273
1459599129,2cyclictest14313-21grep10:30:243
1459599128,3cyclictest20381-21sed07:20:183
1459599127,3cyclictest28780-21taskset10:58:423
14595991210,1cyclictest11191-21ssh10:10:003
1459599120,3cyclictest0-21swapper/309:25:433
1459099129,2cyclictest9940-21head11:45:182
1459099129,2cyclictest8377-21tr11:25:152
1459099129,2cyclictest8238-21tail11:05:202
1459099129,2cyclictest5931-21tr09:05:142
1459099129,2cyclictest32462-21fgrep11:00:222
1459099129,2cyclictest24111-21wc07:30:002
1459099129,2cyclictest20453-21head12:10:262
1459099129,2cyclictest18892-21grep07:15:272
1459099129,2cyclictest18281-21head11:50:172
1459099129,2cyclictest12383-21grep09:50:242
1459099128,3cyclictest4737-21sort07:55:162
1459099128,3cyclictest13767-21turbostat08:15:012
1459099128,3cyclictest13171-21tr10:30:162
1459099126,2cyclictest225-21systemd-journal12:33:462
1459099122,8cyclictest24566-21kworker/2:008:20:002
1459099120,1cyclictest141rcu_preempt12:03:222
1459099120,11cyclictest28599-21tail07:35:252
1459099120,11cyclictest0-21swapper/212:05:022
1458799129,2cyclictest7806-21diskmemload12:30:241
1458799129,2cyclictest7246-21grep08:00:171
1458799129,2cyclictest3155-21chrt11:21:341
1458799129,2cyclictest28395-21grep08:45:161
1458799129,2cyclictest26878-21sed09:20:251
1458799129,2cyclictest19822-21tr07:20:161
1458799129,2cyclictest16977-21grep09:15:141
1458799129,2cyclictest16977-21grep09:15:141
1458799129,2cyclictest12558-21grep12:25:211
1458799127,4cyclictest38150irq/131-snd_hda10:20:491
1458799121,1cyclictest0-21swapper/110:18:381
1458299129,2cyclictest7304-21grep09:05:240
1458299129,2cyclictest3811-21cron07:55:010
1458299129,2cyclictest31955-21grep07:45:150
1458299129,2cyclictest29203-21sed10:00:270
1458299129,2cyclictest28806-21grep10:20:140
1458299129,2cyclictest25061-21ssh09:20:120
1458299129,2cyclictest16278-21sed10:50:280
1458299129,2cyclictest16144-21cron08:20:000
1458299128,3cyclictest6661-21ssh11:24:190
1458299126,2cyclictest13-21ksoftirqd/009:58:110
1458299123,6cyclictest17779-21apache_accesses09:35:120
14582991212,0cyclictest0-21swapper/012:10:520
14582991210,1cyclictest12404-21grep08:10:160
14582991210,1cyclictest0-21swapper/008:25:140
1458299120,12cyclictest0-21swapper/011:39:040
1459599119,1cyclictest7391-21grep09:05:263
1459599119,1cyclictest14274-21ssh10:10:543
1459599118,2cyclictest5147-21ssh10:44:493
1459599118,2cyclictest29999-21cut07:40:173
1459599118,2cyclictest28456-21ssh10:20:113
1459599118,2cyclictest27831-21chrt08:42:123
1459599118,2cyclictest2703-21cat07:50:203
1459599118,2cyclictest25149-21expr11:35:143
1459599118,2cyclictest230692sleep308:32:183
1459599118,2cyclictest20632-21chrt08:26:293
1459599118,2cyclictest19953-21ssh11:50:513
1459599118,2cyclictest16659-21grep08:20:163
1459599118,2cyclictest16153-21tune2fs11:10:193
1459599118,2cyclictest14169-21taskset09:32:353
1459599118,2cyclictest13597-21ssh11:09:033
1459599118,2cyclictest10647-21ssh12:24:543
1459599116,3cyclictest26707-21ssh09:40:183
1459599110,7cyclictest0-21swapper/310:39:063
1459099118,2cyclictest388-21ssh10:03:122
1459099118,2cyclictest304-21sort11:20:152
1459099118,2cyclictest2997-21taskset08:59:482
1459099118,2cyclictest25684-21perf08:40:002
1459099118,2cyclictest21959-21sed10:15:262
1459099118,2cyclictest20306-21tail07:20:182
1459099118,2cyclictest19500-21sh10:52:582
1459099118,2cyclictest18591-21grep09:35:182
1459099118,2cyclictest16050-21grep11:10:172
1459099110,10cyclictest17578-21grep08:20:212
1458799119,1cyclictest3372-21tail07:50:281
1458799118,2cyclictest6499-21sh10:06:201
1458799118,2cyclictest5313-21grep07:55:211
1458799118,2cyclictest3593-21tr09:00:141
1458799118,2cyclictest3216-21ssh11:02:291
1458799118,2cyclictest18158-21sort11:50:171
1458799118,2cyclictest18049-21ssh11:10:591
1458799118,2cyclictest17365-21uname08:20:201
1458799118,2cyclictest14979-21sed08:15:201
1458799118,2cyclictest12921-21wc08:10:211
1458799118,2cyclictest12883-21sh10:30:131
1458799118,2cyclictest12402-21sh11:27:161
1458799117,3cyclictest27178-21tr07:35:141
1458799111,2cyclictest225-21systemd-journal07:40:011
1458299119,1cyclictest2634-21sed08:55:250
1458299119,1cyclictest2503-21head09:25:230
1458299118,2cyclictest499-21ssh10:41:140
1458299118,2cyclictest32180-21wc08:50:250
1458299118,2cyclictest29355-21cat08:45:210
1458299118,2cyclictest27500-21uname09:40:240
1458299118,2cyclictest27466-21ls11:55:240
1458299118,2cyclictest27255-21ssh10:57:300
1458299118,2cyclictest22733-21grep07:25:170
1458299118,2cyclictest20302-21ssh10:34:400
1458299118,2cyclictest18005-21sleep12:10:000
1458299118,2cyclictest17538-21grep08:20:200
1458299118,2cyclictest16082-21ssh11:49:200
1458299118,2cyclictest12271-21cat09:50:250
1458299115,3cyclictest5451-21systemctl07:55:220
1458299110,6cyclictest5447-21perf09:05:000
1458299110,10cyclictest8012-21ssh11:25:120
1459599108,2cyclictest9680-21cut08:05:173
1459599108,1cyclictest50782sleep309:00:433
1459599108,1cyclictest28334-21taskset08:45:143
1459599108,1cyclictest27741-21expr10:00:153
1459599108,1cyclictest27316-21ssh11:55:213
1459599108,1cyclictest14929-21tr07:10:153
1459599107,2cyclictest4633-21df07:55:143
1459599107,2cyclictest29037-21cron07:40:013
1459599107,2cyclictest20317-21rm09:35:513
1459599107,2cyclictest14158-21grep08:15:153
1459599104,4cyclictest28419-21systemctl12:15:233
1459099108,1cyclictest25664-21sendmail_mailqu07:30:222
1459099107,2cyclictest6272-21ssh10:06:102
1459099107,2cyclictest27598-21grep08:40:282
1458799108,1cyclictest9628-21sed08:05:161
1458799108,1cyclictest28332sleep108:57:361
1458799108,1cyclictest21929-21taskset07:25:031
1458799108,1cyclictest20458-21tail08:25:271
1458799108,1cyclictest11720-21sh10:10:141
1458799108,1cyclictest0-21swapper/110:40:151
1458799107,2cyclictest31944-21tr07:45:151
1458799104,5cyclictest15423-21cut07:10:181
1458299108,2cyclictest30775-21rm12:36:020
1458299108,2cyclictest10886-21ssh09:10:520
1458299108,1cyclictest8687-21chrt08:03:380
1458299108,1cyclictest2661-21ssh12:20:030
1458299108,1cyclictest2523-21ssh11:01:550
1458299108,1cyclictest24304-21chrt07:30:000
1458299108,1cyclictest22420-21awk08:30:210
1458299108,1cyclictest21712-21munin-run07:25:000
1458299107,2cyclictest518-21ssh11:40:050
1458299107,2cyclictest26738-21sed08:40:180
145959998,1cyclictest13426-21chrt08:10:463
145909997,1cyclictest21641-21cat08:30:142
145829998,1cyclictest23916-21cat08:35:160
145829997,1cyclictest14788-21chrt07:10:090
145829994,2cyclictest0-21swapper/007:15:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional